-
Notifications
You must be signed in to change notification settings - Fork 16
feat: [Orchestration] Support Fallback Modules #743
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| val requestConfig = | ||
| OrchestrationConfig.create().modules(moduleConfigs).stream( | ||
| configs[0].getGlobalStreamOptions()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The stream options are taken from the first config only. (I don't think there is a way to change this.)
orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationClient.java
Outdated
Show resolved
Hide resolved
CharlesDuboisSAP
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java
Outdated
Show resolved
Hide resolved
| Grounding.create() | ||
| .filters( | ||
| DocumentGroundingFilter.create() | ||
| .dataRepositoryType(DataRepositoryType.HELP_SAP_COM)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Grounding in testFallbackModules?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added this here to be able to test that the additional configs (like grounding or input filter) are correctly propagated when using the fallback modules. Since this is not an e2e test, this will not add flakyness.
Context
AI/ai-sdk-java-backlog#342
This PR introduces support for using fallback modules when using orchestration chat completion. Instead of a single
OrchestrationModuleConfigobject, users can now add multiple of these module configs to a chat completion call (streaming or non-streaming). The configs are tried in the order they are given and if a call with a config failed, the next in line is used as fallback.This PR also introduces a convenience layer to do this:
In the above example, the call with
brokenConfigwill return an error so another call withworkingConfigis performed and the (positive) result of this call is returned.Feature scope:
Definition of Done
Aligned changes with the JavaScript SDK