Metal backend: test modules #17076

manuelcandales · 2026-01-31T00:25:37Z

This pull request adds new support for building and testing the Metal backend modules on macOS, including automation in the CI workflow and a dedicated test runner script. The main changes are the addition of a reusable shell script for building and running Metal backend tests, and the integration of a new GitHub Actions job to automate module testing for the Metal backend.

CI/CD and Testing Automation:

Added a new test-metal-backend-modules job to the .github/workflows/metal.yml workflow, which runs on a macOS M2 runner, sets up the environment, builds the Metal runtime, and runs Python unit tests for the Metal backend modules.

Tooling and Scripts:

Introduced a new script backends/apple/metal/tests/run_metal_test.sh that provides commands to build the Metal runtime, check if it is built, and run inference with given model files. This script is now used in the CI workflow to automate Metal backend testing.

[ghstack-poisoned]

manuelcandales · 2026-01-31T00:25:39Z

Stack from ghstack (oldest at bottom):

pytorch-bot · 2026-01-31T00:25:42Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17076

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 12 Cancelled Jobs, 1 Unrelated Failure

As of commit 0834659 with merge base ba6de95 ():

NEW FAILURES - The following jobs have failed:

pull / unittest / macos / macos-job (gh)
export/tests/test_target_recipes.py::TestTargetRecipes::test_linear_model
pull / unittest-buck / linux / linux-job (gh)
RuntimeError: Command docker exec -t e7c1ef052d8ef793aeb0fe759eccc1d28547b918f3a419c376b7d682de9a4ed0 /exec failed with exit code 3
Test CUDA Builds / test-model-cuda-e2e (mistralai, Voxtral-Mini-3B-2507, quantized-int4-tile-packed) / linux-job (gh)
RuntimeError: Command docker exec -t a5d9e53c4e589657588c030c0c4aee5c45b3dd6d2a9e32f5bbbb8b0a56493aba /exec failed with exit code 1

CANCELLED JOBS - The following jobs were cancelled. Please retry:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest-buck / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 3

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: c8aec0d ghstack-comment-id: 3826704927 Pull-Request: #17076

github-actions · 2026-01-31T00:26:20Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot

Pull request overview

This PR adds comprehensive test coverage for the Metal backend by introducing module-level export and runtime execution tests.

Changes:

Adds test_modules.py with 27 test modules covering operations like matrix multiplication, convolution, attention (SDPA), normalization, and composite blocks
Adds run_metal_test.sh build script to compile Metal runtime and run inference tests
Extends CI workflow with test-metal-modules job that builds Metal runtime and runs module tests

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 12 comments.

File	Description
backends/apple/metal/tests/test_modules.py	New comprehensive test suite with 27 model modules tested across float32 and bfloat16, including export validation and runtime output consistency checks
backends/apple/metal/tests/run_metal_test.sh	New bash script to build Metal runtime with CMake and run inference with executor_runner binary
.github/workflows/metal.yml	Adds new CI job to build Metal runtime and run module tests on macos-m2-stable runner

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

backends/apple/metal/tests/test_modules.py

backends/apple/metal/tests/run_metal_test.sh

backends/apple/metal/tests/test_modules.py

[ghstack-poisoned]

ghstack-source-id: e853016 ghstack-comment-id: 3826704927 Pull-Request: #17076

[ghstack-poisoned]

ghstack-source-id: 8ff1636 ghstack-comment-id: 3826704927 Pull-Request: #17076

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

backends/apple/metal/tests/test_modules.py

.github/workflows/metal.yml

Copilot · 2026-01-31T03:07:22Z

backends/apple/metal/tests/test_modules.py

+EXECUTOR_RUNNER = BUILD_DIR / "executor_runner"
+RUN_METAL_TEST_SCRIPT = TESTS_DIR / "run_metal_test.sh"
+
+# Test output directory - use current working directory in CI for reliable write access


The comment mentions 'aoti_debug_data' directory but doesn't explain why this specific directory name is used or what 'aoti' stands for (Ahead-Of-Time Inference). Consider adding a comment explaining that this directory name is used for Metal backend test outputs to maintain consistency with AOTI conventions.

Suggested change

# Test output directory - use current working directory in CI for reliable write access

# Test output directory - use current working directory in CI for reliable write access.

# We use the 'aoti_debug_data' directory name here to keep Metal backend test outputs

# consistent with Ahead-Of-Time Inference (AOTI) debug data conventions used elsewhere.

[ghstack-poisoned]

ghstack-source-id: c36fc89 ghstack-comment-id: 3826704927 Pull-Request: #17076

mergennachin

What are your thoughts on reusing Backend Test Harness?

https://github.com/pytorch/executorch/tree/main/backends/test/suite

test_modules.py almost look like this:

https://github.com/pytorch/executorch/tree/main/backends/test/suite/operators

I think we should try to reuse as much as possible, and if possible contribute back to test harness

cc @GregoryComer

manuelcandales · 2026-02-02T19:12:02Z

What are your thoughts on reusing Backend Test Harness?

Every backend has its own tests, independent of our more comprehensive testing frameworks. E.g. xnnpack, vulkan, qualcomm, arm

The intent of frameworks like FACTO or Backend Test Harness is to find unknown issues, on mature operators/backends where certain level of coverage and correctness is expected.

On the other hand, having the ability to add specific tests remains useful, as we develop and want to verify that a given PR intended to enable certain feature, is indeed providing the desired functionality.

It is also a minimum bar that the backend/operator should pass, and that will be enforced on CI.

mergennachin · 2026-02-02T20:20:33Z

Discussed offline with @manuelcandales

Agreed that we need both approaches (backend specific unit testing, and backend test harness)

As discussed, let's remove unnecessary tests that's not testing what you're developing right now and could be enabled by backend test harness in the future.

[ghstack-poisoned]

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

backends/apple/metal/tests/test_modules.py

[ghstack-poisoned]

larryliu0820 · 2026-02-04T17:36:27Z

backends/apple/metal/tests/run_metal_test.sh

+    CMAKE_ARGS="-DEXECUTORCH_BUILD_METAL=ON \
+                -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
+                -DEXECUTORCH_BUILD_EXECUTOR_RUNNER=ON \
+                -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
+                -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
+                -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
+                -DEXECUTORCH_BUILD_EXTENSION_NAMED_DATA_MAP=ON \
+                -DAOTI_METAL=ON \
+                -DEXECUTORCH_LOG_LEVEL=Info \
+                -DCMAKE_BUILD_TYPE=Release"


why not use preset? cmake --workflow --preset llm-metal-stats

If not llm specific, also strongly recommend to add a base preset metal-release and let llm-metal-stats extend it.

larryliu0820 · 2026-02-04T17:39:56Z

backends/apple/metal/tests/run_metal_test.sh

+    echo "Running inference..."
+    echo "  PTE: $pte_path"
+
+    "$EXECUTOR_RUNNER" --model_path "$pte_path"


it would be much better to add pybind and run tests in pure python environment. This way you don't need this script.

Update

39db621

[ghstack-poisoned]

manuelcandales requested a review from shoumikhin as a code owner January 31, 2026 00:25

Copilot AI review requested due to automatic review settings January 31, 2026 00:25

manuelcandales requested a review from cccclai as a code owner January 31, 2026 00:25

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 31, 2026

manuelcandales added a commit that referenced this pull request Jan 31, 2026

Metal backend: test modules

760b4bf

ghstack-source-id: c8aec0d ghstack-comment-id: 3826704927 Pull-Request: #17076

Copilot started reviewing on behalf of manuelcandales January 31, 2026 00:25 View session

manuelcandales removed request for cccclai and shoumikhin January 31, 2026 00:26

manuelcandales requested review from larryliu0820 and mergennachin January 31, 2026 00:26

Copilot AI reviewed Jan 31, 2026

View reviewed changes

Update

0ed7c5c

[ghstack-poisoned]

manuelcandales added a commit that referenced this pull request Jan 31, 2026

Metal backend: test modules

83028a8

ghstack-source-id: e853016 ghstack-comment-id: 3826704927 Pull-Request: #17076

Update

b4310cc

[ghstack-poisoned]

Copilot AI review requested due to automatic review settings January 31, 2026 03:01

manuelcandales added a commit that referenced this pull request Jan 31, 2026

Metal backend: test modules

478743b

ghstack-source-id: 8ff1636 ghstack-comment-id: 3826704927 Pull-Request: #17076

Copilot started reviewing on behalf of manuelcandales January 31, 2026 03:01 View session

Copilot AI reviewed Jan 31, 2026

View reviewed changes

Update

94c823c

[ghstack-poisoned]

manuelcandales added a commit that referenced this pull request Jan 31, 2026

Metal backend: test modules

9305ec8

ghstack-source-id: c36fc89 ghstack-comment-id: 3826704927 Pull-Request: #17076

mergennachin reviewed Jan 31, 2026

View reviewed changes

mergennachin requested a review from GregoryComer January 31, 2026 15:04

Update

31b6f45

[ghstack-poisoned]

Copilot AI review requested due to automatic review settings February 2, 2026 21:27

This was referenced Feb 2, 2026

Metal backend: enable linear with bias #17115

Open

Metal backend: remove parakeet linear decomp #17116

Open

Metal backend: support 4-bit linear #17117

Open

Metal backend: enable quantization in test_modules #17118

Open

Copilot AI reviewed Feb 2, 2026

View reviewed changes

backends/apple/metal/tests/test_modules.py Outdated Show resolved Hide resolved

backends/apple/metal/tests/test_modules.py Show resolved Hide resolved

backends/apple/metal/tests/test_modules.py Show resolved Hide resolved

backends/apple/metal/tests/test_modules.py Show resolved Hide resolved

mergennachin approved these changes Feb 2, 2026

View reviewed changes

Update

0834659

[ghstack-poisoned]

This was referenced Feb 2, 2026

Metal backend: test 4-bit linear with bias #17127

Open

Metal backend: int4 refactor checks and logs #17186

Open

Metal backend: import qmv_impl from MLX #17187

Open

Metal backend: fix bugs in qmv_impl #17188

Open

larryliu0820 reviewed Feb 4, 2026

View reviewed changes

-# Test output directory - use current working directory in CI for reliable write access
+# Test output directory - use current working directory in CI for reliable write access.
+# We use the 'aoti_debug_data' directory name here to keep Metal backend test outputs
+# consistent with Ahead-Of-Time Inference (AOTI) debug data conventions used elsewhere.

Metal backend: test modules #17076

Are you sure you want to change the base?

Metal backend: test modules #17076

Conversation

manuelcandales commented Jan 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manuelcandales commented Jan 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jan 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17076

❌ 3 New Failures, 12 Cancelled Jobs, 1 Unrelated Failure

Uh oh!

github-actions bot commented Jan 31, 2026

This PR needs a release notes: label

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 31, 2026

Choose a reason for hiding this comment

Uh oh!

mergennachin left a comment

Choose a reason for hiding this comment

Uh oh!

manuelcandales commented Feb 2, 2026

Uh oh!

mergennachin commented Feb 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

larryliu0820 Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

larryliu0820 Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

larryliu0820 Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

manuelcandales commented Jan 31, 2026 •

edited

Loading

manuelcandales commented Jan 31, 2026 •

edited

Loading

pytorch-bot bot commented Jan 31, 2026 •

edited

Loading

This PR needs a `release notes:` label