MCU hardware benchmark compared to TFLite

[Faboulou]

Hello,

I’m currently evaluating ExecuTorch (Tag V1.0.0 export & Runner) vs TFLite on real hardware (not FVP).

The goal is purely technology validation for industrial use cases, before selecting a runtime for deployment on microcontroller-class devices.

Target: Cortex-M55 + Ethos-U55

Model: MobileNetV2 (int8, Ethos-U compatible)

For ExecuTorch: from torchvision.models import mobilenet_v2

For TFLite: tf.keras.applications.MobileNetV2

The same model architecture and equivalent quantization were used for both runtimes (both with INT8 input/output).

ExecuTorch

PTE size: 3.5 MB (3 512 288 bytes)

Memory:

• Planned (with Input / Output): 256 KiB
• Runtime / temporary: 1.6 MiB
• Method: 30 KiB

Total :
256 + 1600 + 30 = 1886 KiB
≈ 1.84 MiB

After run:
Runtime (used): 1.44 MiB (1474 KiB)
Method (used): 0.24 KiB

Total real:
1464 + 0.24 + 256 = 1720 KiB
= 1.679 MiB

Inference time:

• One image: 113 ms
• 100 images: 113.748 ms

Timing only on:
g_method->execute();

---

TensorFlow Lite

Vela TFLite size: 3.7 MB (3 684 976 bytes)

Arena memory before running (with inputs/outputs): 1.7 MiB

Arena used (after run): 1.51 MiB

Inference time:

• One image: 91.221 ms
• 100 images: 91.204 ms

Timing only on:
g_interpreter->Invoke();

As I understand it, when comparing memory usage, the TFLite arena is roughly equivalent to Planned + Runtime.

Do these values seem correct to you?
Based on these measurements, TFLite currently shows lower latency and memory usage than ExecuTorch, which was unexpected on our side and would be interesting to understand.

cc @psiddh @AdrianLundell @digantdesai @psiddh

[rascani]

Hi @Faboulou!
Thank you for submitting this. We are actually in the process of doing similar benchmarking of MobileNetV2 with real hardware (#16628). The ExecuTorch support for Cortex-M was very limited in 1.0, but has since been expanded with the recently released 1.1. However, I would have expected MV2 to entirely be partitioned to Ethos-U. Would you be able to share more details about how you exported the model? In particular, I'm curious about the Ethos-U configuration.

[gggekov]

Hi @Faboulou ,
For the ExecuTorch execution, do you run mv2 with FP32 input tensor, Quantize to int8, Ethos custom delegate, Dequantize to FP32 (default ExecuTorch behaviour), or do you run with the input & output tensors in int8 ? The execution of the Q/DQ nodes would cost some cycles in the case of ExecuTorch. For TFLite, the default behaviour is int8 inputs/outputs whereas in ExecuTorch, the default behaviour is FP32 inputs/outputs.

You can apply the ExecuTorch QuantizeInputs / QuantizeOutputs passes to produce a pte directly with int8 input/output which will save CPU cycles - i think that would be a closer comparison between TFLite vs ExecuTorch.

You can also run with FP32 inputs/outputs and use the CMSIS-NN Q/DQ implementations.
CC @AdrianLundell

[Faboulou]

Hi @rascani

Sure, this is how I exported the model :

mobilenet_v2 = mobilenetv2.mobilenet_v2(
    weights=mobilenetv2.MobileNet_V2_Weights.IMAGENET1K_V1
).eval()

example_inputs = (torch.randn(1, 3, 224, 224),)

compile_spec = EthosUCompileSpec(
    target="ethos-u55-64",
    system_config="custom",
    memory_mode="Shared_Sram",
    extra_flags=[
        "--output-format=raw",
        "--debug-force-regor",
    ],
    config_ini="/Vela/custom.ini",
)

graph_module = torch.export.export(mobilenet_v2, example_inputs).module(
    check_guards=False
)
_ = graph_module.print_readable()

quantizer = EthosUQuantizer(compile_spec)
operator_config = get_symmetric_quantization_config(is_per_channel=True)
quantizer.set_global(operator_config)


quantized_graph_module = prepare_pt2e(graph_module, quantizer)
quantized_graph_module(*example_inputs)
quantized_graph_module = convert_pt2e(quantized_graph_module)
_ = quantized_graph_module.print_readable()

quantized_exported_program = torch.export.export(quantized_graph_module, example_inputs)
partitioner = EthosUPartitioner(compile_spec)

edge_program_manager = to_edge_transform_and_lower(
    quantized_exported_program,
    partitioner=[partitioner],
    compile_config=EdgeCompileConfig(
        _check_ir_validity=False,
    ),
)

passes = [
    QuantizeInputs(edge_program_manager, [0], method_name="forward"),
    QuantizeOutputs(edge_program_manager, [0], method_name="forward"),
]

edge_program_manager = edge_program_manager.transform(passes)

executorch_program_manager = edge_program_manager.to_executorch(
    config=ExecutorchBackendConfig(
        extract_delegate_segments=False,
        memory_planning_pass=MemoryPlanningPass(
            alloc_graph_input=True, alloc_graph_output=True
        ),
    )
)

print("\nFinal Graph")
_ = executorch_program_manager.exported_program().graph_module.print_readable()

output_path = "mv2_ethos_u55.pte"
save_pte_program(executorch_program_manager, output_path)

---

However, I would have expected MV2 to entirely be partitioned to Ethos-U.

what do you mean ? This means to be the case

CPU operators = 0 (0.0%)
NPU operators = 116 (100.0%)

from :

Network summary for out
Accelerator configuration                Ethos_U55_64
System configuration                           custom
Memory mode                               Shared_Sram
Accelerator clock                                 400 MHz
Design peak SRAM bandwidth                       2.98 GB/s
Design peak Off-chip Flash bandwidth             0.05 GB/s

Total SRAM used                               1474.34 KiB
Total Off-chip Flash used                     3406.42 KiB

CPU operators = 0 (0.0%)
NPU operators = 116 (100.0%)

Average SRAM bandwidth                           0.63 GB/s
Input   SRAM bandwidth                          15.21 MB/batch
Weight  SRAM bandwidth                          14.37 MB/batch
Output  SRAM bandwidth                           8.38 MB/batch
Total   SRAM bandwidth                          39.70 MB/batch
Total   SRAM bandwidth            per input     39.70 MB/inference (batch size 1)

Average Off-chip Flash bandwidth                 0.05 GB/s
Input   Off-chip Flash bandwidth                 0.00 MB/batch
Weight  Off-chip Flash bandwidth                 3.33 MB/batch
Output  Off-chip Flash bandwidth                 0.00 MB/batch
Total   Off-chip Flash bandwidth                 3.33 MB/batch
Total   Off-chip Flash bandwidth  per input      3.33 MB/inference (batch size 1)

Neural network macs                         300836992 MACs/batch

[Faboulou]

Hello @gggekov,

No, as you can see in the export code, I ran QuantizeInputs and QuantizeOutputs, so the model is fully int8.

Both models in my benchmark use full int8 for inputs and outputs.

[gggekov]

Thanks @Faboulou - yes, you are with int8 input/output tensors.
Just to double check - are you using the latest ExecuTorch 1.1 with the corresponding version of the compiler(Vela 4.5.0) ? Run $ vela --version to check that.

The next thing to do is to dig a bit more into what is the NPU doing in the two scenarios. Can I please ask you to poke the Ethos-U PMU counters and I will help you with the interpretation of the results. For TFLite & ExecuTorch, can you please display the following PMU events:

• ETHOSU_PMU_NPU_ACTIVE (total number of NPU Active cycles)
• ETHOSU_PMU_AXI0_RD_DATA_BEAT_RECEIVED (number of beats read via AXI0 interface of the NPU- the SRAM)
• ETHOSU_PMU_AXI0_WR_DATA_BEAT_WRITTEN (number of beats written via AXI0)
• ETHOSU_PMU_AXI1_RD_DATA_BEAT_RECEIVED (number of beats read via the AXI1 interface of the NPU)
• ETHOSU_PMU_MAC_ACTIVE (number of cycles when the MAC array is active)
• ETHOSU_PMU_WD_ACTIVE (number of cycles when the Weight Decoder is active)
  The Ethos PMU mask allows a maximum of 4 PMU events per execution, so you would need two executions for the two frameworks to gather the 6 PMU counters above.

See executorch/examples/arm/executor_runner/arm_perf_monitor.cpp for inspiration how to read the PMU counters.

Also, a more general comment - I would say that for a proper apples to apples comparison between TFLite & ET, you have to start with the same PyTorch model, benchmark it in ExecuTorch, then convert the PyTorch workload to TFLite(e.g. with litert-torch or ONNX or another conversion tool) and benchmark the converted workload in TFLite. Otherwise, the mv2 implementation of PyTorch and TFLite will certainly be different, the workloads will contain different number of MACs, one network may finish with a softmax and the other one with a linear, etc.

[gggekov]

Also, please include the Vela summary for the TFLite & ExecuTorch executions. In both frameworks, the network lowers fully to the NPU, I am interested in the Neural network macs field.

[Faboulou]

Hey @gggekov !

Ok, I will update to ExecuTorch 1.1.0 and Vela 4.5.0, then relaunch the benchmark.

I’ll ping you once it’s done with the PMU events + torch mv2 convert to TFlite .

---

Right now, my setup (for Python export + embedded system runner) is ExecuTorch 1.0.0 with Vela 4.4.1.

Just in case you’re interested in the current Vela report:

For now, here’s the Vela summary for the ExecuTorch executions.

Network summary for out
Accelerator configuration                Ethos_U55_64
System configuration                           custom
Memory mode                               Shared_Sram
Accelerator clock                                 400 MHz
Design peak SRAM bandwidth                       2.98 GB/s
Design peak Off-chip Flash bandwidth             0.05 GB/s

Total SRAM used                               1474.34 KiB
Total Off-chip Flash used                     3406.42 KiB

CPU operators = 0 (0.0%)
NPU operators = 116 (100.0%)

Average SRAM bandwidth                           0.63 GB/s
Input   SRAM bandwidth                          15.21 MB/batch
Weight  SRAM bandwidth                          14.37 MB/batch
Output  SRAM bandwidth                           8.38 MB/batch
Total   SRAM bandwidth                          39.70 MB/batch
Total   SRAM bandwidth            per input     39.70 MB/inference (batch size 1)

Average Off-chip Flash bandwidth                 0.05 GB/s
Input   Off-chip Flash bandwidth                 0.00 MB/batch
Weight  Off-chip Flash bandwidth                 3.33 MB/batch
Output  Off-chip Flash bandwidth                 0.00 MB/batch
Total   Off-chip Flash bandwidth                 3.33 MB/batch
Total   Off-chip Flash bandwidth  per input      3.33 MB/inference (batch size 1)

Neural network macs                         300836992 MACs/batch

---

For now Vela summary for TFLite executions

===== VELA OUTPUT =====


Network summary for mobilenetv2_int8
Accelerator configuration                Ethos_U55_64
System configuration                           custom
Memory mode                               Shared_Sram
Accelerator clock                                 400 MHz
Design peak SRAM bandwidth                       2.98 GB/s
Design peak Off-chip Flash bandwidth             0.05 GB/s

Total SRAM used                               1474.22 KiB
Total Off-chip Flash used                     3597.34 KiB

CPU operators = 0 (0.0%)
NPU operators = 96 (100.0%)

Average SRAM bandwidth                           0.36 GB/s
Input   SRAM bandwidth                          12.67 MB/batch
Weight  SRAM bandwidth                          12.02 MB/batch
Output  SRAM bandwidth                           6.67 MB/batch
Total   SRAM bandwidth                          31.53 MB/batch
Total   SRAM bandwidth            per input     31.53 MB/inference (batch size 1)

Average Off-chip Flash bandwidth                 0.04 GB/s
Input   Off-chip Flash bandwidth                 0.00 MB/batch
Weight  Off-chip Flash bandwidth                 3.31 MB/batch
Output  Off-chip Flash bandwidth                 0.00 MB/batch
Total   Off-chip Flash bandwidth                 3.32 MB/batch
Total   Off-chip Flash bandwidth  per input      3.32 MB/inference (batch size 1)

Neural network macs                         304451664 MACs/batch

The first remark will be the difference MAC counts and the number of NPU operators, but the Torch-to-TFLite conversion may perform graph transformations that yield equivalent results.

I’m not sure how significant this effect is here — curious to hear your thoughts.

[gggekov]

Just to confirm - the TFLite Vela summary is for a PyTorch model that you have then converted to TFLite with one of the conversion tools ?

but the Torch-to-TFLite conversion may perform graph transformations that yield equivalent results.

It's hard to tell without knowing the transformation that the conversion tool is doing and that's why this comparison between TFLite vs ExecuTorch is far from trivial. E.g. if you take a naive PyTorch to TFLite conversion tool where you get a Transpose for every operator to pass from channels-first(PyTorch) to channels-last(TF) around every operator, the TFLite graph will be highly unoptimised because every Transpose will be a fallback to CPU and ExecuTorch will be a lot quicker. At the same time, as per my message above, the mv2 implementation of PyTorch and TFLite will most likely be slightly different making it tricky to do the apples to apples comparison. It would be interesting to inspect the PMU counters for the two frameworks to get a sense for why one is performing slightly better than the other.

[Faboulou]

Just to confirm - the TFLite Vela summary is for a PyTorch model that you have then converted to TFLite with one of the conversion tools ?

No — this one comes from a Keras model. I’ll send you the Vela summary for the PyTorch model converted to TFLite soon.

I only shared this one because I already had the log available and thought it could be useful for comparing it with the PyTorch-to-TFLite Vela summary that I’ll send shortly.