Fix CI Divergence (Mostly `mypy`) by jeff-hykin · Pull Request #1293 · dimensionalOS/dimos

jeff-hykin · 2026-02-18T21:32:10Z

Why does the new CI workflow have a ARM/MacOS runner?
Why are there modifications to pyproject.toml?

Mypy divergence has been a miserable rabbit hole, so please bear with me
whether mypy fails/passes depends on EVERY version of EVERY dependency (because it inspects their source code)
- Easy fix right? uv sync --freeze <ci stuff>
- no. Because of the <ci stuff>
In principle uv sync <ci stuff> shouldn't fail on any system -- b/c in our pyproject toml if something only works on X system we should adding conditionals like "drake; only if linux". But currently we don't because we don't have any checks (and not everyone should get a jetson and a mac just to test their changes to pyproject.toml work on all systems)
This PR should prevent those pyproject.toml issues from getting on dev
Once uv sync <ci stuff> runs, our local mypy won't diverge from CI

Why is `setup.py` modified?

Turns out, when testing with the macos-15 github runner, there's a MacOS install issue with dimos. On MacOS the python executable can be "universal" (one executable for x86 and arm). The python executable for the uv python action on github runners is universal. In our setup.py, we add C-compiler optimization arguments (e.g. -march=native). This causes uv sync to break because clang can only perform that optimization if it targets specifically arm64 xor x86 MacOS (can't do it when targeting for MacOS's universal executable format). The setup.py keeps the optimization whenever possible. For people who use nix, they get the optimization.

Why is there a `fast-docs.yml`?

Right now when I push doc changes, nothing actually checks the doc links or that diagrams are up to date (see https://github.com/dimensionalOS/dimos/actions/runs/22152132097/job/64046102043?pr=1291 for an example). This should fix that and test diagrams (which I don't think we were checking before)

…M machines

This reverts commit dd90ef4.

…s into jeff/fix/ci

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

…s into jeff/fix/gen-diagrams

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

jeff-hykin · 2026-02-18T22:38:26Z

bin/gen-diagrams

@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+set -euo pipefail


Merge #1291 (very small) first and this diff will go away

jeff-hykin · 2026-02-18T22:41:14Z

bin/pr-check

@@ -0,0 +1,99 @@
+#!/usr/bin/env python3


This is what everyone should run locally before making a pull request.
It

checks branch name (you can ignore the error and keep going)

runs only doclink checks if only markdown is edited

runs bin/ci-fast if more than just markdown is edited

A lot of good enhancements could be added, but I wanted to start simple

Why just the fast tests? Shouldn't people run all the tests? (The non-fast ones are more commonly broken on dev especially since a few of them don't run in CI at all.)

jeff-hykin · 2026-02-18T22:44:02Z

bin/pr-name-check

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash


I don't care which branch-naming-scheme we use, I just implemented a check for the one that is listed on our wiki <name>/<type>/<description>

Right now its not a forceful/blocking check it'll just yell at you if it doesnt match

jeff-hykin · 2026-02-18T22:47:17Z

docker/python/Dockerfile

 # Install dependencies with UV (10-100x faster than pip)
 RUN uv pip install --upgrade 'pip>=24' 'setuptools>=70' 'wheel' 'packaging>=24' && \
-    uv pip install '.[misc,cpu,sim,drone,unitree,web,perception,visualization]'
+    uv sync --frozen --extra misc --extra visualization --extra agents --extra web --extra perception --extra unitree --extra manipulation --extra cpu --extra dev --extra sim --extra drone --extra base


Most Important Change

--frozen instead of "close enough" install

An issue I see here is that this forces us to always run with uv run .... This should be fixable by adding some of the variables which activating gives you. This could be enough:

ENV VIRTUAL_ENV=/app/.venv ENV PATH="/app/.venv/bin:$PATH"

Another issue that worries me is that uv sync always installs in .venv. The previous command installed in the system python. By installing in /app it means we can't mount our local dir as /app because it would use the .venv we have locally instead of the one in the docker image.

Not sure what the solution is here. (Maybe copy pyproject.toml to /app-deps and install from that dir so the packages are installed outside /app?

jeff-hykin · 2026-02-18T22:47:52Z

docs/development/writing_docs.md

    - If it only matters to people who contribute to dimos (like this doc), put them in `docs/development`
    - Otherwise put them in `docs/usage`
-2. Run `bin/gen_diagrams` to generate the svg's for your diagrams. We use [pikchr](https://pikchr.org/home/doc/trunk/doc/userman.md) as a diagram language.
+2. Run `bin/gen-diagrams` to generate the svg's for your diagrams. We use [mermaid](https://mermaid.js.org/intro/) (no generation needed) and [pikchr](https://pikchr.org/home/doc/trunk/doc/userman.md) as diagrams languages.


Merge #1291 (very small) first and this diff will go away

jeff-hykin · 2026-02-18T23:12:54Z

.github/workflows/docker.yml

      cmd: "MYPYPATH=/opt/ros/humble/lib/python3.10/site-packages mypy dimos"
      dev-image: ros-dev:${{ (needs.check-changes.outputs.python == 'true' || needs.check-changes.outputs.dev == 'true' || needs.check-changes.outputs.ros == 'true') && needs.ros-dev.result == 'success' && needs.check-changes.outputs.branch-tag || 'dev' }}

-  # Run module tests directly to avoid pytest forking issues


If we need to restore old code, lets use git as it was meant to be (instead of having giant blocks of commented out code

jeff-hykin · 2026-02-18T23:13:27Z

.github/workflows/docker.yml

    with:
-      cmd: "pytest -m heavy"
+      cmd: "uv run pytest -m heavy"
      dev-image: dev:${{ (needs.check-changes.outputs.python == 'true' || needs.check-changes.outputs.dev == 'true') && needs.dev.result == 'success' && needs.check-changes.outputs.branch-tag || 'dev' }}


I think this is a nessary change, based on: https://github.com/dimensionalOS/dimos/actions/runs/22160016061/job/64077594237?pr=1293

jeff-hykin · 2026-02-18T23:20:36Z

pyproject.toml

    # Planning (Drake)
-    "drake>=1.40.0; platform_machine != 'aarch64'",
+    "drake==1.45.0; sys_platform == 'darwin' and platform_machine != 'aarch64'",
+    "drake>=1.40.0; sys_platform != 'darwin' and platform_machine != 'aarch64'",


This is both a fix for MacOS and Arm. On MacOS drake 1.45.0 is needed, on x86 arm anything >1.40 is okay.

The drake team says they're interested in supporting Arm linux so hopefully that'll come soon

jeff-hykin · 2026-02-18T23:21:52Z

pyproject.toml

 cpu = [
    # CPU inference backends
-    "onnxruntime",
+    "onnxruntime<1.24; python_version < '3.11'",


CI caught this which is great: it was impossible for python 3.10 to have the version of onnx we had specified in the lock file. This guard fixes that

jeff-hykin · 2026-02-18T23:23:11Z

setup.py

 from setuptools import find_packages, setup

+
+def python_is_macos_universal_binary(executable: str | None = None) -> bool:


without this change macos-15 github uv python runner can't uv sync

jeff-hykin · 2026-02-18T23:25:34Z

@greptile-apps please re-review. Edit your original message if you are able to

greptile-apps

_{15 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

.github/workflows/fast-docs.yml

.github/workflows/fast-test.yml

paul-nechifor · 2026-02-19T03:13:01Z

bin/ci-slow

+
+echo
+echo "== dev: pytest -m integration =="
+run_in_image "dev" "pytest -m integration"


If you're running all of them sequentially, you might as well run a single command:

run_in_image "dev" "-m 'not (tool or module or neverending or mujoco)'"

bin/pr-name-check

Co-authored-by: Paul Nechifor <paul@nechifor.net>

paul-nechifor · 2026-02-19T03:50:27Z

.github/workflows/docker.yml

    secrets: inherit
    with:
-      cmd: "pytest && pytest -m ros" # run tests that depend on ros as well
+      cmd: "uv run pytest && uv run pytest -m ros" # run tests that depend on ros as well


By changing environment variables we can make the uv python be the default python so that we don't need to prefix with uv run (mentioned in a comment below)

paul-nechifor · 2026-02-19T03:53:33Z

bin/ci-fast

+echo "# pytest fast"
+echo "# "
+echo "# "
+time uv run bin/pytest-fast


pytest-fast has this:

. .venv/bin/activate exec pytest "$@" dimos

You probably just want to run pytest without any -m, not ./bin/pytest-fast.

My thought was, if we edit pytest-fast (to exclude something, or add something) we would want that change to be in sync with CI.

paul-nechifor · 2026-02-19T03:59:11Z

bin/ci-fast

+set -euo pipefail
+
+echo "Ensuring pip versions are correct"
+time uv sync --frozen --extra misc --extra visualization --extra agents --extra web --extra perception --extra unitree --extra manipulation --extra cpu --extra dev --extra sim --extra drone --extra base


I'm not sure this should be here. In CI, the packages should have been installed in the python step, no?

And locally this removes packages. (I do uv sync --all-extras locally.)

No, AFAIK the github uv python action doesn install with --frozen which is what is needed to prevent mypy from diverging. Just run the workflow and you'll see this command does cause installs.

Running the command when everything is already installed is almost instant (no reinstall, just performs a check).

As for locally, its still basically necessary. I agree uninstallation isnt ideal but I don't think theres a way to say "keep all the features I selected (dont uninstall anything) but use the frozen versions from UV". If you dont want to run the command thats fine but IMO we should at least have one command that exactly replicates CI

I can try to make a "use frozen versions of everything but dont uninstall anything" command. If its not too hacky maybe that would be useful

leshy · 2026-02-19T08:02:52Z

.github/workflows/fast-docs.yml

+        with:
+          python-version: '3.12'
+
+      - name: Run doclinks # Note: doesn't need uv python, just needs some python to run the script


git-commit hooks run this, so this is already executed by our code-cleanup action in CI

I was thinking we would change it where this is the only thing that would run on .md file edits (to make it really fast to merge doc changes). If we keep code cleanup though, yeah I guess it would be good to remove it.

I guess there are those whitespace checks in pre-commit that still effect docs

spomichter · 2026-02-20T07:29:10Z

@jeff-hykin there is no MacOS runner

jeff-hykin · 2026-02-20T15:12:09Z

@jeff-hykin there is no MacOS runner

dimos/.github/workflows/fast-test.yml

Line 23 in 25aedba

os: [ubuntu-24.04, macos-15, ubuntu-24.04-arm]

its right here? I ran it on my own repo to make sure it worked on MacOS too

jeff-hykin · 2026-02-20T15:25:28Z

I need to figure out whats going on with mypy not existing on the docker side.

…f/fix/ci

jeff-hykin and others added 30 commits February 10, 2026 09:28

add arm support

98725c5

-

8696799

Update dependency specifications for open3d to support ARM and non AR…

5913387

…M machines

Merge branch 'dev' into jeff/arm_support

3470deb

patch for CI

cf097a2

fix mypy errors

17fabf8

Fix: conditionally include xcbuild for Darwin platforms only

87702f4

Fix: conditionally exclude packages without ARM wheels

f6e6794

format

3318e91

try to make mypy CI happy

9c4ef0f

remove unused

37f05dc

Merge branch 'dev' into jeff/arm_support

d4f53ca

Merge branch 'dev' into jeff/arm_support

bf457f1

basic CI test

bd3b49d

only run on python changes

dd90ef4

Revert "only run on python changes"

bda7800

This reverts commit dd90ef4.

clean up

7b2309d

test

5d70fac

-

366599e

Merge branch 'dev' into jeff/fix/ci

5d58aaa

Merge branch 'dev' into jeff/fix/ci

aa763eb

add a fast docs checker

ce0110e

add missing command

a5cff27

Merge branch 'jeff/fix/gen-diagrams' of github.com:dimensionalOS/dimo…

fefcf26

…s into jeff/fix/ci

Apply suggestion from @greptile-apps[bot]

f254a3c

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

fix

aaa4e98

Merge branch 'jeff/fix/gen-diagrams' of github.com:dimensionalOS/dimo…

028429a

…s into jeff/fix/gen-diagrams

Update bin/gen-diagrams

1056faa

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

-

7c29734

Merge branch 'dev' into jeff/arm_support

ba7be91

jeff-hykin commented Feb 18, 2026

View reviewed changes

jeff-hykin changed the title ~~Fix CI Divergence~~ Fix CI Divergence (Mostly mypy) Feb 18, 2026

jeff-hykin added 2 commits February 18, 2026 17:08

fixup minor aspects

e0849db

Merge branch 'dev' of github.com:dimensionalOS/dimos into jeff/fix/ci

7d1cd3e

jeff-hykin commented Feb 18, 2026

View reviewed changes

greptile-apps bot reviewed Feb 18, 2026

View reviewed changes

.github/workflows/fast-docs.yml Outdated Show resolved Hide resolved

.github/workflows/fast-test.yml Show resolved Hide resolved

.github/workflows/fast-test.yml Outdated Show resolved Hide resolved

only install brew if needed

1be6ecf

paul-nechifor reviewed Feb 19, 2026

View reviewed changes

bin/pr-name-check Outdated Show resolved Hide resolved

paul-nechifor reviewed Feb 19, 2026

View reviewed changes

bin/pr-name-check Show resolved Hide resolved

jeff-hykin and others added 2 commits February 18, 2026 21:33

Update bin/pr-name-check

214e748

Co-authored-by: Paul Nechifor <paul@nechifor.net>

Update bin/pr-name-check

25aedba

Co-authored-by: Paul Nechifor <paul@nechifor.net>

paul-nechifor reviewed Feb 19, 2026

View reviewed changes

leshy reviewed Feb 19, 2026

View reviewed changes

jeff-hykin added 2 commits February 20, 2026 11:37

Merge branch 'jeff/fix/ci' of github.com:dimensionalOS/dimos into jef…

d06c169

…f/fix/ci

-

91b9a00

		from setuptools import find_packages, setup


		def python_is_macos_universal_binary(executable: str \| None = None) -> bool:

Comments

Conversation

jeff-hykin commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why does the new CI workflow have a ARM/MacOS runner? Why are there modifications to pyproject.toml?

Why is setup.py modified?

Why is there a fast-docs.yml?

Uh oh!

jeff-hykin Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paul-nechifor Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeff-hykin Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Most Important Change

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeff-hykin Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeff-hykin commented Feb 18, 2026

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paul-nechifor Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

paul-nechifor Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeff-hykin Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeff-hykin Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leshy Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeff-hykin Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

jeff-hykin commented Feb 18, 2026 •

edited

Loading

Why does the new CI workflow have a ARM/MacOS runner?
Why are there modifications to pyproject.toml?

Why is `setup.py` modified?

Why is there a `fast-docs.yml`?

jeff-hykin Feb 18, 2026 •

edited

Loading

paul-nechifor Feb 19, 2026 •

edited

Loading

jeff-hykin Feb 18, 2026 •

edited

Loading

jeff-hykin Feb 18, 2026 •

edited

Loading

paul-nechifor Feb 19, 2026 •

edited

Loading

paul-nechifor Feb 19, 2026 •

edited

Loading

jeff-hykin Feb 20, 2026 •

edited

Loading

jeff-hykin Feb 20, 2026 •

edited

Loading

leshy Feb 19, 2026 •

edited

Loading

jeff-hykin Feb 20, 2026 •

edited

Loading

jeff-hykin commented Feb 20, 2026 •

edited

Loading