Skip to content

Conversation

@Jonas-Isr
Copy link
Member

@Jonas-Isr Jonas-Isr commented Jan 30, 2026

Context

AI/ai-sdk-java-backlog#342

This PR introduces support for using fallback modules when using orchestration chat completion. Instead of a single OrchestrationModuleConfig object, users can now add multiple of these module configs to a chat completion call (streaming or non-streaming). The configs are tried in the order they are given and if a call with a config failed, the next in line is used as fallback.

This PR also introduces a convenience layer to do this:

var prompt = new OrchestrationPrompt("your prompt");

var workingConfig = new OrchestrationModuleConfig().withLlmConfig(GPT_4O_MINI);
var secondWorkingConfig = new OrchestrationModuleConfig().withLlmConfig(GPT_4O);
var brokenConfig =
    new OrchestrationModuleConfig()
        .withLlmConfig(new OrchestrationAiModel("broken_name", Map.of(), "latest"));

var fallbackConfigs = new OrchestrationModuleConfig[] {workingConfig, secondWorkingConfig};

OrchestrationChatResponse result = client.chatCompletion(prompt, brokenConfig, fallbackConfigs);
// or:
// OrchestrationChatResponse result = client.chatCompletion(prompt, brokenConfig, workingConfig, secondWorkingConfig);

In the above example, the call with brokenConfig will return an error so another call with workingConfig is performed and the (positive) result of this call is returned.

Feature scope:

  • test low-level code
  • add convenience layer
  • unit and e2e tests

Definition of Done

@Jonas-Isr Jonas-Isr self-assigned this Jan 30, 2026
@Jonas-Isr Jonas-Isr added the please-review Request to review a pull-request label Jan 30, 2026
Comment on lines 59 to 61
val requestConfig =
OrchestrationConfig.create().modules(moduleConfigs).stream(
configs[0].getGlobalStreamOptions());
Copy link
Member Author

@Jonas-Isr Jonas-Isr Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stream options are taken from the first config only. (I don't think there is a way to change this.)

Copy link
Contributor

@CharlesDuboisSAP CharlesDuboisSAP left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Grounding.create()
.filters(
DocumentGroundingFilter.create()
.dataRepositoryType(DataRepositoryType.HELP_SAP_COM));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grounding in testFallbackModules?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this here to be able to test that the additional configs (like grounding or input filter) are correctly propagated when using the fallback modules. Since this is not an e2e test, this will not add flakyness.

@Jonas-Isr Jonas-Isr merged commit 68a27ec into main Feb 3, 2026
6 checks passed
@Jonas-Isr Jonas-Isr deleted the orch_fallback_modules branch February 3, 2026 14:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

please-review Request to review a pull-request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants