Do we need to do anything to support merged job/workflow chat?

[josephjclark]

I think it's fine for the workflow chat and job chat to take a mixed history, where different models have produced different responses?

Are there any gotchas I haven't thought about? Any compatibility issues?

See for the frontend work OpenFn/lightning#4109

[josephjclark]

Possible things to do:

• run rag in job chat? When going from workflow -> job or job -> job
• job chat history might actually include history from previous jobs! How do we handle this?

[hanna-paasivirta]

a) RAG

When we go from workflow -> job: RAG (retrieval decision + optional retrieval*) will run automatically because there won't be a rag key in the payload.

If we go from workflow -> job1 -> workflow -> job2: What happens depends on how Lighting calls job_chat for job2. If Lightning constructs the new call based only on the previous Apollo response, then RAG will run as there wouldn't be a RAG key in the previous response.

If we go from job1 -> job2: If Lightning constructs the new call based on the previous Apollo response, then RAG might not run as there might already be a rag key.

More details on RAG:

Summary of how we decide to run RAG in Apollo:
rag not present → RAG runs
rag: null → RAG runs
rag: {} → RAG runs
rag: {data: [...]} → RAG skipped (unless refresh_rag: true)

*Note that the RAG system includes a step to decide whether we actually need to run RAG given the conversation context. This means that it might not fetch documentation every time even if refresh_rag is true.

b) History

I assume Lightning will construct each new call based on the user's message + the previous Apollo response, which always includes history + the either the YAML/job code+adaptor based on what the user is looking at. This is similar to how it works now and will provide all necessary information (except old RAG results, but that's ok).

The conversation history includes suggested YAML but not suggested job code from the model. It does not include the YAML/job code that Lightning sent to Apollo, i.e. user turns are recorded as just the user message.

Problems:

1. the job_chat service currently tries to write YAML if requested and especially when included in the history -> fix prompt
2. there might be inconsistencies that the model can't understand -> fix prompts
3. both services might lack essential information (e.g. "given the workflow I created manually, fill in the code for the first step"; "ok I applied your job code edits, now check my workflow nodes make sense") -> I can adjust prompts slightly but this seems more of a global assistant problem

Also, the conversation history "content" keys in job_chat are plain text, while they are JSON objects in workflow_chat.

job_chat outputs:
{"role": "assistant", "content": "Plain text explanation"}
workflow_chat outputs:
{"role": "assistant", "content": '{"text": "explanation", "yaml": "..."}'}

So there's another potential problem:
4. The model might confuse the answer formats in the history and those requested in the prompt -> add a note in the prompts + test how bad this is

TODO in Lightning

Lightning should trigger a RAG refresh when the user navigates to a different job. Lightning can trigger a RAG refresh any time by dropping an existing rag key or by adding this flag: `refresh_rag: true` #321

TODO in Apollo

Prompt:

• Add note in job_chat and workflow_chat prompts to explain that earlier conversation turns may have had different context & different answer formats, and that the attachments have been removed
• Add a note in job_chat prompt as well to ask the user to navigate to the workflow if relevant. Match to the note in workflow_chat. These could specify that the user can also copy the context into the conversation.

History/RAG:

• Normalise history across the two services: redact workflow_chat history to only have plain text, i.e. remove the nested suggested YAML key
• Use Anthropic structured outputs to help with consistency Use Anthropic structured outputs #310
• Add a new page identifier key (type, name) to history and the latest user message, which will identify what the conversation relates to, and whether we need to run RAG. We should re-run RAG if the current message relates to a job & that job is different from the last message in the history.