Add Tosa to LLM extension by Sebastian-Larsson · Pull Request #15556 · pytorch/executorch

Sebastian-Larsson · 2025-11-04T14:22:26Z

cc @freddan80 @per @zingo @oscarandersson8218 @digantdesai @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng

pytorch-bot · 2025-11-04T14:22:32Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15556

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 9 New Failures, 1 Pending, 2 Unrelated Failures

As of commit f4cf8c0 with merge base b17004c ():

NEW FAILURES - The following jobs have failed:

pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t ded1468a5f2fd64f2eca42df79121e93057ecf3bcb30974f69770ac62c6d5439 /exec failed with exit code 1
pull / unittest / macos / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / unittest-arm-backend-with-no-deps (test_pytest_models_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t 2e4dee7d405da6421869b726ff699183753f234ab54c83c2a71fcfa962d0db2b /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t 08d9902cedd9ee22f069e720f4a7df69033c597f38e01c55c2875242d4bcd82b /exec failed with exit code 1
pull / unittest-editable / macos / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-arm-backend-ethos-u (test_pytest_models_ethos_u55) / linux-job (gh)
RuntimeError: Command docker exec -t 243a76db2f553cf8201be60e24ee4d2417d4e30679e550c604766ed5a8b23f45 /exec failed with exit code 1
trunk / test-arm-backend-ethos-u (test_pytest_models_ethos_u85) / linux-job (gh)
RuntimeError: Command docker exec -t b2879c44072773b87d09734676bf27bdd272ffdd4ce0ac36b41d2cd5f65a52ed /exec failed with exit code 1
trunk / unittest-release / linux / linux-job (gh)
RuntimeError: Command docker exec -t df79e6eeb31c113817e4b4b083111923655ac5a69089bc04396b77e17740f2f3 /exec failed with exit code 1
trunk / unittest-release / macos / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

trunk / test-models-windows (emformer_transcribe, portable) / windows-job (gh) (detected as infra flaky with no log or failing log classifier)
trunk / test-models-windows (mobilebert, portable) / windows-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

zingo · 2025-11-11T07:47:43Z

Hi @SS-JIA this one is outside of Arm stuff so we need help with someone to review it, thanks for your time and effort with this.

SS-JIA · 2025-11-11T20:21:11Z

Requested review from @larryliu0820

larryliu0820 · 2025-11-11T21:17:00Z

examples/models/llama/export_llama_lib.py

            llm_config.quantization.pt2e_quantize.value
        )
        quantizers.append(coreml_quantizer)
+    if llm_config.backend.tosa.enabled and llm_config.quantization.pt2e_quantize:


Is it possible to add a unit test to cover this logic branch?

Let me know what you think

Sebastian-Larsson · 2025-12-02T08:23:42Z

@larryliu0820 woul you mind taking another look with the changes after your comments?

zingo · 2025-12-04T08:59:50Z

@metascroy Is this OK to merge?

zingo · 2025-12-15T12:03:05Z

Hi @larryliu0820 (and/or @digantdesai ) is this OK after the added test? We have some follow up PR that would be good to get into PR state :)

SS-JIA · 2026-01-07T16:53:18Z

@larryliu0820 new test case was added - mind taking a look? Thanks in advance!

SS-JIA

LGTM as the requested test was added!

SS-JIA

Temporary RC while I double check internal importing the diff

meta-codesync · 2026-01-14T17:31:51Z

@SS-JIA has imported this pull request. If you are a Meta employee, you can view this in D90692071.

SS-JIA · 2026-01-14T20:50:37Z

Indeed, the new test is failing due to the Arm targets not being visible to the test target.

To fix, update: https://github.com/pytorch/executorch/blob/8293fa64d221c1d694723b9c209fe0eeb34a8748/examples/models/llama/tests/TARGETS

Add the "//executorch/backends/arm/quantizer:lib" target as a dependency

larryliu0820 · 2026-01-20T18:00:49Z

Looks good as long as CI jobs are fixed.

zingo · 2026-01-29T12:53:57Z

Rebasing as we just merged a tosa spec change just to make sure all is still working :)

zingo · 2026-01-30T13:44:57Z

@SS-JIA we fixed the buck2 things and now it needs to be re-imported as we cant merge with the version diff vs meta internal.

Change-Id: I2cded64d7ffdf3de441bbd7ee73357c8ba0a3749 Signed-off-by: Sebastian Larsson <sebastian.larsson@arm.com>

Sebastian-Larsson · 2026-02-03T13:32:47Z

Updated from TARGETS to BUCK

zingo · 2026-02-03T14:13:37Z

@SS-JIA Can you help reimporting? And maybe review if it's green?

SS-JIA · 2026-02-03T22:08:34Z

@zingo apologies for the delay. I triggered a re-import but still seeing some failures internally - currently trying to figure out what is needed to get everything green, and will post an update ASAP

Summary: cc freddan80 per zingo oscarandersson8218 digantdesai larryliu0820 mergennachin cccclai helunwencser jackzhxng Pull Request resolved: pytorch#15556 Differential Revision: D90692071 Pulled By: SS-JIA

SS-JIA · 2026-02-03T23:25:46Z

@Sebastian-Larsson @zingo I created #17183 to demonstrate the fixes are that needed to get things working in Meta-internal repo!

To minimize the land time, I think we can either:

Stamp / land the PR I created - I can take care of any further fixes needed for internal tests to pass
Incorporate the changes from the PR I created into this one, I'll re-import and double check tests pass

I do notice some arm related failures on this PR, though - I presume those are unrelated to these changes, and there are no other planned updates to this PR?

Sebastian-Larsson · 2026-02-04T10:20:51Z

@Sebastian-Larsson @zingo I created #17183 to demonstrate the fixes are that needed to get things working in Meta-internal repo!

To minimize the land time, I think we can either:
* Stamp / land the PR I created - I can take care of any further fixes needed for internal tests to pass

* Incorporate the changes from the PR I created into this one, I'll re-import and double check tests pass
I do notice some arm related failures on this PR, though - I presume those are unrelated to these changes, and there are no other planned updates to this PR?

Landing the PR you created seems like the smoother option if you don't mind!

This diff is to facilitate the landing of #15556 Differential Revision: [D92280949](https://our.internmc.facebook.com/intern/diff/D92280949/) [ghstack-poisoned]

This diff is to facilitate the landing of #15556 Differential Revision: [D92280949](https://our.internmc.facebook.com/intern/diff/D92280949/) ghstack-source-id: 338222346 Pull Request resolved: #17205

SS-JIA · 2026-02-04T15:30:15Z

Of course. I'll take it from here!

Sebastian-Larsson requested review from jackzhxng and larryliu0820 as code owners November 4, 2025 14:22

Sebastian-Larsson added the partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm label Nov 4, 2025

Sebastian-Larsson requested review from lucylq and mergennachin as code owners November 4, 2025 14:22

Sebastian-Larsson added ciflow/trunk release notes: examples Changes to any of our example LLMs integrations, such as Llama3 and Llava module: llm Issues related to LLM examples and apps, and to the extensions/llm/ code labels Nov 4, 2025

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 4, 2025

larryliu0820 reviewed Nov 11, 2025

View reviewed changes

Sebastian-Larsson force-pushed the tosa_llm_ext branch from 2efc03e to c4d72e5 Compare November 20, 2025 12:21

Sebastian-Larsson mentioned this pull request Dec 17, 2025

Add ethos-u and vgf to LLM extension #16288

Open

zingo added this to ExecuTorch Arm Backend Dec 19, 2025

github-project-automation bot moved this to To triage in ExecuTorch Arm Backend Dec 19, 2025

zingo added this to the 1.1.0 milestone Dec 19, 2025

hth000 mentioned this pull request Jan 13, 2026

Accuracy drop : DeiT-tiny & LLaMA behavior differences between FP / Decomposition / FQ / TOSA #16426

Open

SS-JIA approved these changes Jan 14, 2026

View reviewed changes

SS-JIA requested changes Jan 14, 2026

View reviewed changes

zingo removed this from the 1.1.0 milestone Jan 28, 2026

zingo moved this from To triage to In progress in ExecuTorch Arm Backend Jan 28, 2026

Sebastian-Larsson force-pushed the tosa_llm_ext branch from 8293fa6 to 9f93123 Compare January 29, 2026 10:02

zingo approved these changes Jan 29, 2026

View reviewed changes

zingo requested a review from SS-JIA January 29, 2026 10:27

zingo added this to the 1.2.0 milestone Feb 2, 2026

Add Tosa to LLM extension

f4cf8c0

Change-Id: I2cded64d7ffdf3de441bbd7ee73357c8ba0a3749 Signed-off-by: Sebastian Larsson <sebastian.larsson@arm.com>

Sebastian-Larsson force-pushed the tosa_llm_ext branch from c974788 to f4cf8c0 Compare February 3, 2026 13:31

SS-JIA mentioned this pull request Feb 4, 2026

Tosa to LLM extension - buck fixes #17183

Open

SS-JIA pushed a commit that referenced this pull request Feb 4, 2026

[executorch][arm] Add Tosa to LLM extension

309784c

This diff is to facilitate the landing of #15556 Differential Revision: [D92280949](https://our.internmc.facebook.com/intern/diff/D92280949/) [ghstack-poisoned]

SS-JIA mentioned this pull request Feb 4, 2026

[executorch][arm] Add Tosa to LLM extension #17205

Merged

Conversation

Sebastian-Larsson commented Nov 4, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15556

❌ 9 New Failures, 1 Pending, 2 Unrelated Failures

Uh oh!

zingo commented Nov 11, 2025

Uh oh!

SS-JIA commented Nov 11, 2025

Uh oh!

larryliu0820 Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Sebastian-Larsson Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Sebastian-Larsson commented Dec 2, 2025

Uh oh!

zingo commented Dec 4, 2025

Uh oh!

zingo commented Dec 15, 2025

Uh oh!

SS-JIA commented Jan 7, 2026

Uh oh!

SS-JIA left a comment

Choose a reason for hiding this comment

Uh oh!

SS-JIA left a comment

Choose a reason for hiding this comment

Uh oh!

meta-codesync bot commented Jan 14, 2026

Uh oh!

SS-JIA commented Jan 14, 2026

Uh oh!

larryliu0820 commented Jan 20, 2026

Uh oh!

zingo commented Jan 29, 2026

Uh oh!

zingo commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sebastian-Larsson commented Feb 3, 2026

Uh oh!

zingo commented Feb 3, 2026

Uh oh!

SS-JIA commented Feb 3, 2026

Uh oh!

SS-JIA commented Feb 3, 2026

Uh oh!

Sebastian-Larsson commented Feb 4, 2026

Uh oh!

SS-JIA commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Sebastian-Larsson commented Nov 4, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Nov 4, 2025 •

edited

Loading

zingo commented Jan 30, 2026 •

edited

Loading