Release v0.0.10: Manipulation Stack, MuJoCo Simulation, DDS Transport, Web and Native Visualization via Rerun#1336
Merged
spomichter merged 105 commits intomainfrom Feb 21, 2026
Merged
Release v0.0.10: Manipulation Stack, MuJoCo Simulation, DDS Transport, Web and Native Visualization via Rerun#1336spomichter merged 105 commits intomainfrom
spomichter merged 105 commits intomainfrom
Conversation
* adding timestamp knowledge to agent prompt and info * Update dimos/perception/experimental/temporal_memory/temporal_utils/graph_utils.py
* added HardwareComponent defining universal schema fo robot type and joint type classes * removed simple HardwareConfig and replaced with new HardwareComponent dataclass * updated blueprints and test files * added JointState dataclass and JointName str alias for better readability * renamed control orchestrator to control coordinator as its easier to say * Fix typo in component schema docstring * added explicit type alias for JointName in ControlTask * added Type Checking import
* add readme * remove unneeded pip install commands * add feedback * add warning * fix exmaples link * enhance message about humancli, add crash warning * add dev/contributing docs * add license * add uvx run * consolidate package usage into readme.md * consolidate and finish development docs * remove redundant doc * docs reorganization pt1 * remove probably outdated docs, consolidate remaining * standardize naming * typo and linking * linking fix * cleanup todo's * move viz docs * add note to dimos run * alignment * rename main dev docs * disable dev container until fixed * minor naming * add links to concepts * fix broken link * shorten readme * fix links * remove --single-branch to avoid problems with checkout * add debugging * extend architecture * reorganize docs * minimal edit * rice the readme * grammar * formatting * fix examples * change links to reduce change count * improve wording * wording * remove acknowledgements * improve the humancli example * formatting * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * remove print that never runs * wording * example of getting module inputs/outputs * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * switch to dev branch for development * fixup linked files * add blueprints back * fix typo * get Paul's blueprints docs closer to runnable by adding imports * add rpc call and skill examples * better docs for skills * improve contributing * changes for paul * Update README.md * Update README.md * fix broken link * update broken link * Readme changes * readme reorg * rename so that it renders on github * break up writing documentation * add minimal API overview * add table of contents to concepts * add actual doclinks command so the doclinks docs make sense * remove stuff stash didn't like * hide TODO in a comment * add a link to blueprints * minor change * use different examples in readme * revert using different examples * update diagram docs * rename back * fix diagrams * add instruction * revert revert * fix link * improve link * fix link * fix link * remove all but the "Simple DimOS Application" * lots of minor fixes * make docs consistent * add back lcmspy (accidental removal) * better wording * improve REPO_ROOT * simplify * wording * should be fully runnable in theory, test next * add doclinks command * Revert "add doclinks command" * improve syntax * make blueprints.md executable (tested) * neutralize edits * Revert "neutralize edits" * remove unused name * test against blueprints.md instead of development.md --------- Co-authored-by: Paul Nechifor <paul@nechifor.net> Co-authored-by: stash <pomichterstash@gmail.com>
* autogenerate list of blueprints * swtich order
Add benchmark test cases for DimosROS pubsub, which uses dimos message types with automatic conversion. This complements existing RawROS tests and allows measuring the dimos↔ROS conversion overhead.
* docs: update transports documentation with encode/decode mixins and module examples - Add encode/decode mixins section with PickleEncoderMixin, LCMEncoderMixin, JpegEncoderMixin - Add working example of deploying modules with shared memory transport - Add ROS and DDS to available transports table - Add doclinks CLI entry point to pyproject.toml - Fix webcam.py to import CameraInfo from dimos.msgs (has with_ts method) instead of dimos_lcm * transports work, blueprints moved, data moved * more transport wording changes * doc simplification * linking go2_basic svg correctly * abstraction layer map * alt transport doc * transports.md wrap * blueprints pulled from dev * reintroduced missing data flow svg * fixed broken link * titles fix * ci: run pre-commit on all pushes and PRs
* fix: default to rerun native viewer * Update dimos/dashboard/rerun_init.py Co-authored-by: Paul Nechifor <paul@nechifor.net> * Change default comment --------- Co-authored-by: Paul Nechifor <paul@nechifor.net> Co-authored-by: stash <pomichterstash@gmail.com>
* Still uses Dask by default. * Start without Dask with: ```bash uv run dimos --no-dask --simulation run unitree-go2-agentic ``` * Startup time improvement so far: 60 seconds before, 45 seconds now. (This is measured from running the command till I can instruct it to "start exploring" and it starts moving). * There are a lot of potential improvements we can do. This is the first step.
…1067) * added HardwareComponent defining universal schema fo robot type and joint type classes * removed simple HardwareConfig and replaced with new HardwareComponent dataclass * updated blueprints and test files * added JointState dataclass and JointName str alias for better readability * renamed control orchestrator to control coordinator as its easier to say * Fix typo in component schema docstring * added explicit type alias for JointName in ControlTask * updated sdk backends to have unit conversion constants at the top. Every backend must implement its own conversion based on the sdk. * added registry to auto detect backends and then import * added gripper constants and parameters * changed logger from debug to info for unavailable sdk wrappers * made naming more consistent - chagned from ambiguous 'backend' name for sdk adapter to 'adapter' * updated readmes * fixed ManipulatorAdapter import after rebase * float type annotation for gripper pos * clean up * remove the caching flag so discover() can be called multiple times
* make camera demo actually show up in rerun * ruff
* fix more ruff issues * fix mypy
* Feat: Adding rerun to camera module * Fix: module name + topic_str * CI code cleanup * Fix: rerun_enabled as local variable, not property * Feat: Added pinhole logging to viz in 3d with robot * Feat/fix: Added tf_rerun and fixed camera height * Fix: camera_link wrt base_link transform * Fix: pre-comit errors * CI code cleanup * commit to Trigger CI
* use iec instead of SI units for bandwidth measurement * Nicer (and IEC) unit calculations for benchmarks * converter cleanup * nicer table
…pnet (#1150) These third-party libraries are installed but missing py.typed markers, causing mypy to raise import-untyped errors in addition to import-not-found.
* refactor(pubsub): reorganize module structure and extract encoders - Move implementations to impl/ subdirectory (lcmpubsub, memory, shmpubsub, etc.) - Extract encoder mixins to encoders.py (PubSubEncoderMixin, PickleEncoderMixin, LCMEncoderMixin, JpegEncoderMixin) - Add AllPubSub and DiscoveryPubSub mixins with complementary default implementations - Add GlobPubSub and RegexPubSub marker classes with docstring examples - Update imports across codebase - Add CLAUDE.md to .gitignore * small fix to pass CI tests * fix: update shmpubsub import path after impl/ reorganization * lcm class cleanup * encoder typing fixes * feat: add Glob and regex pattern support to LCM Topic subscriptions - Add Glob class for glob-style pattern matching (*, **, ?) - Topic now accepts str, re.Pattern, or Glob for flexible subscription patterns - Pattern subscriptions return the actual matched channel in callback - Remove duplicate LCMMsg and Topic from lcmservice.py (use DimosMsg) - Add PubSubProtocol for structural typing - Add tests for regex and glob pattern subscriptions * feat: add subscribe_all and Topic.from_channel_str for typed pattern subscriptions - Add subscribe_all() method to LCMPubSubBase for subscribing to all topics - Add Topic.from_channel_str() factory method to parse channel strings with embedded type info - Channel format: /topic#module.ClassName enables automatic type extraction - Add test for subscribe_all with typed message decoding * refactor: extract resolve_msg_type to dimos.msgs.helpers Move message type resolution logic to a dedicated helper module with lru_cache for performance. Supports fallback to dimos_lcm module path. * lcm pattern sub test rename * pattern sub tests and docs * bridge type spec * fix: bridge.py type annotations and missing subscribe_topic param - Fix AllPubSub type parameter order (TopicFrom, MsgFrom not MsgFrom, TopicFrom) - Fix pass_msg callback signature to match spec (MsgFrom, TopicFrom) - Pass subscribe_topic config to bridge() function * fix: resolve mypy type errors across pubsub modules - helpers.py: Add type: ignore for getattr Any return - lcmpubsub.py: Add type: ignore for callback type variance and mixin incompatibility - shmpubsub.py: Add type: ignore for mixin incompatibility - transport.py: Add arg-type to existing type: ignore * pattern sub test fix * pattern tests fixed * fix: add import-untyped to mypy ignore comments for cupy/contact_graspnet These third-party libraries are installed but missing py.typed markers, causing mypy to raise import-untyped errors in addition to import-not-found. * fix: add DecodingError and LCM_SELF_TEST filtering to pubsub encoders - Add DecodingError exception for skipping messages in decode() - Add LCMTopicProto protocol for type-safe LCM topic handling - Filter LCM_SELF_TEST topic in both encoders and lcmpubsub handler - Fix type annotations in shmpubsub and ros_bridge * fix: resolve ruff warnings for explicit re-exports and loop variable binding - Use explicit re-export syntax in impl/__init__.py - Bind loop variable in lambda default argument to fix B023
* fix: LCMPubSubBase.subscribe accepts Topic only, not Topic | str * fix: add thread-safety locks to subscribe_new_topics and subscribe_all * fix: improve type safety in pubsub and skill comms - LCMCommsConfig now uses LCMTopic with default_factory - Add proper type params to PubSubComms and LCMSkillComms - Fix SkillMsg type annotations with MsgType param - Add explicit re-exports in protocol/service/__init__.py - Wrap string topic in LCMTopic in pLCMTransport * fix: remove type ignores from PubSubBaseMixin - Add subscribe method declaration to mixin - Fix _Subscription type to use PubSubBaseMixin
* switch to create pattern * rename ModuleBlueprint to _BlueprintAtom, and ModuleBlueprintSet to Blueprint to be consistent with docs * add protocol helpers * fix lingering name changes * add get_protocol_method_signatures * rename ModuleConnection to Stream to align with docs * add ModuleRef to _BlueprintAtomic, and name StreamRef for consistency * organize rerun within build * add missing type hints for RPCClient * remove completely unused variable * add missing type hints * fix arg * improve type hinting for RPCClient * avoid using builtin name as variable name * fixup ModuleProxy * add _connect_module_refs * formatting * fix typing of static methods and args * use stable python interfaces * fix annotation-protocol in the pytoml * fix type annotations * fix test * update tests * one char test fix * fixup typing * backport changes that were made in future branches * fix start/stop test * fix naming * get module refs working! * fix test * fix basic tests * fixup typing * CI code cleanup * fix mypy * fix mypy * fix class checks * fix weird merge issues * rename "connections" to "streams" * fix isclass check * update docs, rename example * ignore typing issues * fix * add is_module check for test --------- Co-authored-by: jeff-hykin <17692058+jeff-hykin@users.noreply.github.com>
…#1079) * added robot_description folders to data/ folder with lfs tracking * Base types: RobotModelConfig, Obstacle, Protocol specs for defining manipulation scenes * Mesh conversion utils for converting any urdf to drake compatible mesh files * DrakeWorld implementation that maintains all objects, robots in the scene and owns the scene graph * Monitor system keeps the world model (drake planning world) in synb with real world. manages obstacle lifecycle and more * FK and IK solver implementation with drake * RRT path planner using default drake planner * factory for building implementation of the different world, viz, planner etc specs * utils for ik solver and path planning * manipulation module manages the world monitor, planner, kinematics planner and interfaces with other modules * added blueprints and manipulation client for testing * added unit test, e2e tests and readme * general cleanup * added multi robot management and control to the manipulation module * refactored planning stack * Refactored manipulation planning stack: seperated planners from world implementation, split kinematics into JacobianIK and DrakeOptimizationIK * updated README * Address greptile comments: manipulation_blueprint - Added optional add_gripper: bool = True to make xarm6 ans xarm7 config consistent. world_obstacle_monitor - Added warning log when obstacle not found during cleanup manipulation_client - Removed unused numpy impor path_utils - Added explicit tolerances atol=1e-6, rtol=0 to np.allclose() for stricter joint-space duplicate detection jacobian_ik - Added division-by-zero protection for velocity limits using nonzero_mask to skip zero-valued limits drake_world.py - added more specific exception handling (avoids hiding bugs) mesh_utils - regex fixes * fix mypy import error with manipulation interface (file is deprecated) * protocol now only requires solve() method. * moved to using standard LCM data types and added type alias for readability * updated all coordinator references * fixed all bluepirnts order * updated paths to use Path object instead of str * mypy type fixes * all public api boundaries use standard message types, sim specific internal storage/helpers still use NDArrays in some places * removed unused import * removed manipulation history reference * added a Jacobian typealias * fixed mypy errors * updated manipulation dependencies in pyproject.toml
* old numpy forced old version of rerun * fix other mypy issues
* fix(unitree-go2): minor issues * fix(unitree-go2): minor issues * fix stand down/up * fix mypy
- Remove direnv install section and wget .envrc pattern from ubuntu/osx/nix guides - Use plain 'source .venv/bin/activate' everywhere - Condense library usage to 4 lines: mkdir, venv, activate, install - Keep nix.md's nix-specific setup (nix develop) but drop direnv from it too
* initial commit * lazyloader to defer heavy imports * Increase test timeout to prevent flaky failures * Fix lcmspy global totals test flaking from LCM discovery packets * Timeout to 15s * lazy imports not necessary * added manipulation to docker/python/ * Fix: default timeout is 30s * lcmspy test to filter per-topic * Fix: skip pydrake type stubs in mypy * Fix: delete .pyi files that breaks mypy on 3.10 in CI * Fix: find all .pyi and delete them * adding pydrake to the ignore_missing_imports override * follow_imports skip for pydrake
…tructions docs(install): remove direnv, simplify installation instructions
…lation (#1309) * fix(realsense): lazy-load pyrealsense2 to avoid import errors in simulation pyrealsense2 is a system-level dependency that requires hardware libraries. Importing it eagerly in camera.py caused ModuleNotFoundError when loading any blueprint from manipulation_blueprints.py, even simulation blueprints that don't use RealSense cameras. - Use __getattr__ in realsense/__init__.py for lazy package exports - Defer pyrealsense2 import to runtime methods (start, _build_camera_info, _intrinsics_to_camera_info, _get_extrinsics) via local imports - Use TYPE_CHECKING guard + from __future__ import annotations for type hints Python's module cache makes repeated local imports effectively free. * fix(data): add pre-generated xarm7.urdf to xarm_description LFS archive (#1312) The keyboard-teleop-xarm7 blueprint references xarm7.urdf via LfsPath, but only xarm7.urdf.xacro existed. CartesianIKTask/PinocchioIK need a plain URDF — they don't process xacro. xarm6.urdf was already pre-generated; xarm7 was missed. Generated from xarm_device.urdf.xacro with dof=7, limited=true using DimOS's own xacro processor (with package_paths resolution). * fix(zed): add zed_camera stub when SDK is not installed The else branch (no pyzed) was missing a zed_camera stub, causing ImportError when blueprints tried to import it. Added a stub that raises ModuleNotFoundError with name='pyzed' so the integration test skips gracefully. Also added pyzed to OPTIONAL_DEPENDENCIES in test_all_blueprints.py. * fix(zed): add type annotations to zed_camera stub
cyclonedds requires a source build with CYCLONEDDS_HOME set, which fails on standard dev machines. Updated install docs to use `uv sync --all-extras --no-extra dds` and added a note to the dds extra in pyproject.toml.
Both were leftovers from the dead Detic integration (import commented out, directory excluded from mypy). Zero imports of either package anywhere in the codebase. Removing them eliminates slow source builds and 3 transitive packages from uv sync.
LCM autoconf prompts for system configuration during test collection, which fails without -s since pytest captures stdin by default.
Switch default viewer_backend from 'rerun' (native) to 'rerun-web' for broader compatibility without a native viewer install.
* add venv setup to dev install docs * nix - both uses .venv
* docs(readme): restore banner image from main * docs(readme): remove duplicate SVG banners
* rerun config g1 * fix: fillmode warning
* fix: invisible due to background colors * default value
* fix(g1): add video stream and camera_info to G1SimConnection G1SimConnection was missing color_image and camera_info outputs, so MuJoCo-rendered frames never reached the rerun bridge. Mirrors the Go2 connection pattern: subscribes to video_stream(), publishes camera_info in a background thread, and adds camera_optical TF frame. * fix(g1): remove /g1/ topic prefix to match Go2 and rerun overrides The rerun visual_overrides from #1334 used world/camera_info and world/color_image (matching Go2), but the G1 transports used /g1/ prefixed topics. This mismatch meant the rerun bridge never applied the camera_info pinhole overlay. Remove the /g1/ prefix from color_image and camera_info transports in the primitive blueprint and the SHM blueprint. Now matches Go2 convention and the rerun overrides work correctly. * fix(g1): skip webcam camera_module in simulation mode In sim, G1SimConnection now provides video from MuJoCo. The webcam camera_module in the primitive was also publishing to color_image, causing interleaved webcam + MuJoCo frames. Conditionally skip camera_module when global_config.simulation is True. * fix(g1): use correct MuJoCo camera intrinsics and add thread exit condition - Replace hardcoded 1280x720 Go2 intrinsics with computed values from MuJoCo constants (320x240, FOV=45°) matching MujocoConnection pattern - Add _stop_event to camera_info loop for clean shutdown * fix(sim): increase MuJoCo video render resolution to 1280x720 * fix(g1): revert MuJoCo resolution, use Go2 camera intrinsics for rerun MuJoCo framebuffer can't exceed 640 width without XML config. Reverted render resolution to 320x240. Use the same 1280x720 camera intrinsics as Go2 for rerun display scaling (matches Go2 sim behavior exactly). * fix(sim): increase MuJoCo render resolution to 1280x720 Set offscreen framebuffer size in model XML to support the higher resolution. Previously failed because MuJoCo's default offscreen buffer is only 640 wide. * Revert "fix(sim): increase MuJoCo render resolution to 1280x720" This reverts commit f21d37b.
Contributor
|
Too many files changed for review. ( |
spomichter
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v0.0.10
The Agentive Operating System for Generalist Robotics
Highlights
88+ commits, 20 contributors, 700+ files changed.
The TLDR: a complete manipulation stack, MuJoCo simulation, DDS transport, and a rewritten visualization pipeline. Agents are no longer bolted on top — they're refactored as native modules with direct stream access. The entire ROS message dependency has been removed from core DimOS, and we've added VR, phone, and arm teleoperation stacks. You can now vibecode a pick-and-place task from natural language to motor commands. Installation has been significantly streamlined — no more direnv, simpler setup, and the web viewer is now the default.
🚀 New Features
Simulation
time.sleep).dimos --simulation run unitree-go2(#1035) by @jca0Manipulation
generate_grasps()returns ranked PoseArray. (#1119, #1234) by @JalajShuklaSSTeleoperation
Transports & Infrastructure
subscribe_all()for bridge-style consumers. Topic type encoding in channel strings (/topic#module.ClassName). (#1114) by @leshylcm_encode()when message is already bytes. (#1223) by @leshyNavigation
Visualization
to_rerun()to Rerun viewer. GlobalConfig singleton, web viewer support. Replaces the old rerun initialization system. (#1154) by @leshyAgents
Platform & Hardware
HardwareComponentdataclass withJointState,JointNametype aliases. Backend registry with auto-discovery for SDK adapters. (#1040, #1067) by @mustafab0🔧 Improvements
--no-daskflag. Startup time reduced from ~60s to ~45s. (#1111, #1232) by @paul-nechiforModuleBlueprint→_BlueprintAtom,ModuleBlueprintSet→Blueprint,ModuleConnection→Stream. AddedModuleRef, improved type hints throughout. (#1143) by @jeff-hykinsubscribe_new_topicsandsubscribe_all. Proper type params across pubsub stack. (#1153) by @leshyQuestButtonstoButtonswith generic field names for cross-platform teleop. (#1261) by @ruthwikdasyam./bin/devnow runs the ROS-enabled dev image. (#1170) by @leshycycloneddsrequires a source build, soddsis now excluded fromuv sync --all-extrasby default. (#1318) by @spomichtertofixmarkers (#1216), fixed ruff issues (#1112), removed old README_installation.md (#1101) by @paul-nechifor🐛 Bug Fixes
uv syncfor some macOS systems (#1322) by @jeff-hykin📚 Documentation
🏗️ CI & Build
@pytest.mark.cudafor GPU-dependent tests. (#1220) by @jeff-hykin-sto default addopts for LCM autoconf compatibility. (#1320) by @spomichterModuleBlueprint→_BlueprintAtom,ModuleBlueprintSet→Blueprint,ModuleConnection→Stream(#1143)CudaImageandNumpyImageremoved. Image is now a pure NumPy dataclass. Methods likesolve_pnp,csrt_tracker,from_depth,to_depth_metersremoved. (#1161)to_ros/from_rosconversion methods removed. UseROSTransportinstead. (#1230)dimos.dashboard.rerun_initremoved. UseRerunBridgeModuleor thererun-bridgeCLI. (#1154)unitree_go2→unitree/go2,unitree_g1→unitree/g1. Blueprint names updated. (#1221)--viewer-backend rerunto restore native viewer. (#1324)Quickstart
New Contributors 🎉
Full Changelog: v0.0.9...v0.0.10