feat: Bundles #3331
Draft
feat: Bundles #3331
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…#3321) This pull request introduces significant improvements to the `@streamr/utils` package, focusing on robust browser and Node.js environment support, build configuration enhancements, and codebase modernization. The main changes include a new dual build and export system for browser and Node.js, environment variable handling refactoring, and the replacement of Node.js-specific APIs with cross-platform polyfills. ### Changes **Build and packaging enhancements:** - Added a dual build system using a new `rollup.config.mts` to generate separate browser and Node.js bundles, with appropriate module resolution and aliasing for each environment. (`packages/utils/rollup.config.mts`) - Updated `package.json` to use the `exports` field for conditional exports (browser vs. Node.js), adjusted the `files` array, and added new build scripts and dependencies for Rollup and polyfills. (`packages/utils/package.json`) [[1]](diffhunk://#diff-21f8eb1d7a149d42ab0454ea905d44f624ff1d0aeb6d7a88d75b42962ca6b053L10-R36) [[2]](diffhunk://#diff-21f8eb1d7a149d42ab0454ea905d44f624ff1d0aeb6d7a88d75b42962ca6b053R47-R69) - Introduced a `prebuild` and `postbuild` step to streamline the build process and ensure clean output. (`packages/utils/package.json`) **Cross-platform and polyfill improvements:** - Replaced direct Node.js API usage (e.g., `crypto`, `os`) with cross-platform abstractions and browser polyfills (e.g., `buffer-shim`, `path-browserify`, custom `os` and `env` shims for browsers). (`packages/utils/src/browser/crypto.ts`, `packages/utils/src/browser/env.ts`, `packages/utils/src/browser/os.ts`, `packages/utils/src/exports-browser.ts`, `packages/utils/src/keyToArrayIndex.ts`, `packages/utils/karma.config.ts`) [[1]](diffhunk://#diff-285b8e8fa583afc20697242820c6adb2419720d1c2056c6ecb7d8617060c1a51R1-R23) [[2]](diffhunk://#diff-245bac5c3851ffe8bc18571e59cf298ecb92e78d093c47fb8b1cef8c740174feR1-R5) [[3]](diffhunk://#diff-0106994557e39509eb1269938f670aa46dea2f4ab6e6e0e7c06124b973cfa268R1-R6) [[4]](diffhunk://#diff-394788f4b1559a7010d4d4a1ae7f96a11cb992f358a46090cfb9c1eebfbfac05L1-R1) [[5]](diffhunk://#diff-394788f4b1559a7010d4d4a1ae7f96a11cb992f358a46090cfb9c1eebfbfac05L28-R28) F8929faeL1) - Removed the old `crossPlatformCrypto.ts` in favor of new environment-specific implementations. (`packages/utils/src/crossPlatformCrypto.ts`) **Environment variable handling:** - Refactored all environment variable access to use a unified `env` abstraction, ensuring safe access in both Node.js and browser contexts. (`packages/utils/src/Logger.ts`, `packages/utils/src/browser/env.ts`) [[1]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4R5) [[2]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L21-R36) [[3]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L46-R45) [[4]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L98-R97) [[5]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L117-R116) [[6]](diffhunk://#diff-91ff14206456eacaae372406d303b78f2a1ca01c710b6f0f005a8ee20b2cf50eR1-R19) **Testing and configuration:** - Updated Jest and Karma configurations to support new module aliases and environment-specific code paths. (`packages/utils/jest.config.ts`, `packages/utils/karma.config.ts`) ([packages/utils/jest.config.tsL1-R11](diffhunk://#diff-138963f3d6083e08b3091ffdcd032a655c4738ba0a41476560753dd9f6b1efeaL1-R11), F8929faeL1) **Code modernization and cleanup:** - Updated cryptographic utilities to use new cross-platform imports and types, improving maintainability and browser compatibility. (`packages/utils/src/SigningUtil.ts`, `packages/utils/src/exports.ts`) [[1]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L9-R9) [[2]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L21-L23) [[3]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L61-R57) [[4]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L94-R90) [[5]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L170-R166) [[6]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L202-R200) [[7]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L232-R231) [[8]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L246-R244) [[9]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L260-R258) [[10]](diffhunk://#diff-d700ca1a1d7a0254f811405cb99a0f7eba3b8ec8396aede7362c8c842fbb7a56L56-R56) **Additional minor improvements:** - Added `@streamr/utils` to the Docker build bootstrap step. (`Dockerfile.node`) These changes collectively make the `@streamr/utils` package more robust, easier to consume in different environments, and simpler to maintain. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > Modernizes `@streamr/utils` for first-class browser and Node.js support with dual bundles, conditional exports, and polyfilled APIs. > > - New Rollup config (`packages/utils/rollup.config.mts`) outputs `exports-browser.*` and `exports-nodejs.*`; `package.json` uses conditional `exports` and updated build scripts > - Introduces env/crypto platform shims: `src/nodejs/{env,crypto}.ts` and `src/browser/{env,crypto,os}.ts`; removes `src/crossPlatformCrypto.ts` > - Replaces Node-only APIs with cross-platform imports: `Logger` reads from `@/env`; hashing via `computeMd5`; updated `SigningUtil` to use `@/crypto` > - Adds browser entry `src/exports-browser.ts` (includes `buffer-shim`) and path aliases for both envs; updates Jest/Karma configs and tsconfigs (`tsconfig.{node,browser,jest,karma}.json`) > - Updates `packages/utils/package.json` fields (`main/module/types`, `files`) and dependencies (Rollup, plugins, polyfills) > - Docker build now bootstraps `@streamr/utils` in `Dockerfile.node` > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit b3774ce. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY -->
This pull request refactors and improves utility modules in the codebase, focusing on the `Logger` and `SigningUtil` classes, as well as minor optimizations in binary utilities and error handling. The main changes include a redesign of the logger initialization for better encapsulation, simplification of the signing utility instantiation logic, and minor performance and correctness tweaks. > [!NOTE] > This is just the first step (alongside the recent `utils` refactoring). Other packages will follow to make the whole thing much more convenient to use. ## Changes **Logger improvements:** - Refactored the `Logger` class to encapsulate the root logger initialization in a static method, removing the global `rootLogger` variable and improving encapsulation. The logger now lazily initializes the root logger as needed. [[1]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L33-R68) [[2]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L54-L78) [[3]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L91-R99) - Improved the logger method wrapping logic to ensure consistent logging behavior in browser environments. [[1]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L33-R68) [[2]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L54-L78) **Signing utility simplification:** - Simplified the `SigningUtil.getInstance` method by replacing the static `keyTypeToInstance` map with a switch statement that directly instantiates the correct class for each key type. This removes unnecessary static instances and clarifies the instantiation logic. [[1]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L34-L38) [[2]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L318-L326) **Utility optimizations:** - Updated `binaryUtils.ts` to create new `TextEncoder` and `TextDecoder` instances on each call instead of using shared instances, preventing potential state issues and improving reliability. **Error handling improvements:** - Moved the creation of the logger instance inside the `executeSafePromise` function to ensure a fresh logger is used for each invocation, and removed the unnecessary top-level logger instance. ## Future steps * Propagate the same technique throughout the other packages.
This pull request introduces a new bundling workflow for the `autocertifier-client` package, adding Rollup-based builds for both JavaScript and TypeScript types. It also updates the package configuration to support both CommonJS and ES module outputs, improves build script management, and refines the package's file exports for better distribution. ### Build system integration * Added a new `rollup.config.mts` file to bundle both CommonJS (`exports.cjs`) and ES module (`exports.mjs`) outputs, as well as generate TypeScript type definitions using Rollup and relevant plugins. * Updated `tsconfig.json` to use `module: preserve` and `moduleResolution: bundler` for compatibility with the new bundling setup. ### Package configuration improvements * Modified `package.json` to specify both `main` (CommonJS) and `module` (ESM) entry points, and refined the `files` field to export only the necessary bundle outputs. * Added new build scripts (`postbuild`, `reset-self`) and improved the build pipeline to include cleaning up build artifacts and running Rollup after TypeScript compilation. ### Dependency updates * Added Rollup, plugins for node resolution and type definitions, `tsx`, and `rimraf` as development dependencies to support the new build process.
This pull request refactors the build and packaging process for the `browser-test-runner` package, introducing Rollup for bundling, improving module resolution, and updating various scripts and dependencies. The changes make the package outputs more consistent, enhance compatibility (especially for Electron and Karma), and modernize the codebase with ES module syntax and improved path handling. ## Changes **Build & packaging improvements:** * Added a new `rollup.config.mts` configuration to bundle outputs for CommonJS, ES modules, Electron preload, and type declarations using Rollup. * Updated `package.json` to define `main`, `module`, and `types` entry points, specify more granular files for publishing, and add build scripts (`prebuild`, `postbuild`, `reset-self`). * Added new devDependencies for Rollup, plugins, and build tooling in `package.json`. * Updated `tsconfig.json` to use `"module": "preserve"` and `"moduleResolution": "bundler"` for better compatibility with modern bundlers. **Code modernization & compatibility:** * Switched from CommonJS `require` to ES module `import` syntax in `karma-setup.js` for `expect`. * Improved path resolution in `createKarmaConfig.ts` to use `fileURLToPath` and `import.meta.url` for locating setup and preload files, ensuring compatibility with ESM and Electron environments. [[1]](diffhunk://#diff-ce10c51de1f8215dc5ff71f630122f7f6ed1d768f3c0e83f0978654b3ac3701fR2-R10) [[2]](diffhunk://#diff-ce10c51de1f8215dc5ff71f630122f7f6ed1d768f3c0e83f0978654b3ac3701fL61-R62) * Updated `createWebpackConfig.ts` to explicitly set `timers: false` in the `fallback` object for module resolution.
> [!NOTE] > Looks like #3332 is too much at once - lotta failing tests and what not. I will deal with each of them packages separately. This pull request updates the build and packaging setup for the `@streamr/cdn-location` package. The main improvements are the introduction of a Rollup build process to generate both CommonJS and ES module outputs, refined output file structure, and enhancements to the development workflow. ## Changes **Build system and packaging improvements:** * Added a new `rollup.config.mts` file to configure Rollup to bundle the package into both ES module (`exports.js`) and CommonJS (`exports.cjs`) formats, as well as to generate type declarations (`exports.d.ts`). This setup also marks dependencies as external to avoid bundling them. * Updated `package.json` to point to the new output files, specify both `main` (CommonJS) and `module` (ESM) entry points, and restrict the published files to only the relevant outputs. * Added Rollup and related plugins (`@rollup/plugin-node-resolve`, `rollup-plugin-dts`, `tsx`) as dev dependencies, and introduced a `postbuild` script to run Rollup after TypeScript compilation. Also added a `prebuild` script to clean up build info files. * Updated `tsconfig.json` to use `"module": "preserve"` and `"moduleResolution": "bundler"` for better compatibility with the new bundling process. **Monorepo integration:** * Included `@streamr/cdn-location` in the monorepo bootstrap process in the root `Dockerfile.node`, ensuring it is built alongside other packages.
This pull request refactors the packaging and module setup for the `@streamr/test-utils` package, modernizing its build process and improving TypeScript and module resolution support. The changes include switching to a Rollup-based build, updating exports for better compatibility, consolidating custom matcher types, and cleaning up obsolete files. These updates make the package easier to consume in different environments and improve developer experience. ### Build and packaging improvements * Added a new `rollup.config.mts` configuration and switched the build process to use Rollup for both JavaScript and TypeScript declaration files, including new scripts in `package.json` for prebuild and postbuild steps. [[1]](diffhunk://#diff-a0a6d5226837d528f922340045f16689ae1305f9e13444f02cd5af3c8ca27ebcR1-R99) [[2]](diffhunk://#diff-d7a3fa8faf23a3288372b5fa5679e4f24c0be0dbb8da3c3399315a40fc4b872cL10-R51) [[3]](diffhunk://#diff-d7a3fa8faf23a3288372b5fa5679e4f24c0be0dbb8da3c3399315a40fc4b872cR70-R76) * Updated the `package.json` exports and files fields to provide clear entry points for both ESM and CommonJS consumers, and to expose custom matcher modules directly. ### TypeScript and module resolution enhancements * Improved TypeScript configuration by updating `typesVersions` and `tsconfig.json` to use bundler-style module resolution and by referencing matcher types directly from `customMatchers` instead of a separate file. [[1]](diffhunk://#diff-d7a3fa8faf23a3288372b5fa5679e4f24c0be0dbb8da3c3399315a40fc4b872cL10-R51) [[2]](diffhunk://#diff-aac742816d4fdf853f9ea01d95b5631fc3fcde78870b2bcdde087dbbe4f5f0a1L8-R12) [[3]](diffhunk://#diff-d78d8c40e5585d63387aa1f7cd3c261e2d617035b9b2aff3bf311fd565a72b6bL13-R13) [[4]](diffhunk://#diff-88900faaa35c4ab3f7ca3ff3f6ef788ac93135bb928f89b7aeab37cd3f97a7a1L21-R21) * Removed the obsolete `customMatcherTypes.ts` file and consolidated matcher type declarations into `customMatchers.ts`, simplifying global type augmentation for Jest and Expect. [[1]](diffhunk://#diff-33cb836509b8b233d42419de8f746d81d5e3f1f11b224bb9a0d430c4a724daccL1-L13) [[2]](diffhunk://#diff-46302ca607b0d6e06ec1cf8d6774e378afb302a28a344ee7765e008421e4b392L50-R68) ### Code cleanup and consistency * Updated imports and exports throughout the codebase to use named exports for `customMatchers` and to ensure matcher setup is consistent and direct. [[1]](diffhunk://#diff-818e39864728c9de8dcb73c4efb093d655013cc3cd4e72376ae986e41a94485fL179-R179) [[2]](diffhunk://#diff-d236e791996dddf0961df2f1c25d2c9a8433a4b116f60e7549d07f4fe3bfd61bL1-R1) [[3]](diffhunk://#diff-6d98f04c9981e55112650c9b4b981c328ada96a1c3cc51d285f71f1377e206bbL1-R1) * Removed the legacy `setupCustomMatchers.js` alias file, as setup is now handled via direct exports and improved build outputs.
This pull request introduces significant improvements to the build and packaging process for the `@streamr/geoip-location` package, including the addition of a Rollup build step, updates to the package entry points, and enhancements to development dependencies. These changes ensure better compatibility and distribution of the package in both CommonJS and ES module formats, streamline type definitions, and improve build script reliability. **Build and packaging improvements:** * Added a new Rollup configuration (`rollup.config.mts`) to bundle output files in both ES and CommonJS formats and to generate type definitions, improving compatibility and distribution. * Updated the `package.json` entry points to explicitly specify CommonJS (`main`), ES module (`module`), and type definitions (`types`) outputs; adjusted the `files` field to exclude unnecessary files from the published package. * Enhanced build scripts in `package.json` by adding `prebuild`, `postbuild`, and `reset-self` steps to automate cleaning and bundling, improving reliability and developer experience. **Dependency and configuration updates:** * Added development dependencies for Rollup, its plugins, and related utilities (`@rollup/plugin-node-resolve`, `rollup`, `rollup-plugin-dts`, `tsx`, `rimraf`) to support the new build process. * Updated TypeScript configuration (`tsconfig.json`) to use `moduleResolution: bundler` and `module: preserve`, aligning with the new bundling approach. **Integration:** * Included the `@streamr/geoip-location` package in the Docker build process to ensure it is bootstrapped along with other packages.
This pull request updates the build and packaging pipeline for the `proto-rpc` package to improve module output, type bundling, and cleanup. The changes introduce Rollup for bundling, refine the output structure, and add supporting scripts and dependencies. **Build and packaging improvements:** * Added a new `rollup.config.mts` configuration file to bundle both CommonJS (`exports.cjs`) and ES module (`exports.js`) outputs, as well as generate bundled type definitions (`exports.d.ts`). * Updated `package.json` to point `main` to `./dist/src/exports.cjs`, add a `module` entry for ESM, and restrict the published files to only the bundled outputs. * Added a `postbuild` script to run Rollup after TypeScript compilation, and a `reset-self` script to clean up build artifacts before building. **Dependency updates:** * Added build-related devDependencies: `rollup`, `rollup-plugin-dts`, `@rollup/plugin-node-resolve`, `tsx`, and `rimraf` for improved bundling and cleanup. **TypeScript configuration:** * Updated `tsconfig.json` to use `module: preserve` and `moduleResolution: bundler` to better support the new bundling workflow.
This pull request introduces a major refactor and modernization of the DHT package's build system, module exports, and browser/node environment separation. The changes migrate the package to a dual build (Node.js and browser), introduce a Rollup-based bundling pipeline, clean up legacy scripts, and refactor source code organization and imports for clarity and maintainability. ## Changes The most important changes are: **Build system & package exports modernization:** - Switched to a Rollup-based build process, with a new `rollup.config.mts` that generates separate Node.js and browser bundles (both JS and type declarations), and added relevant plugins and scripts to `package.json`. This replaces the previous postbuild shell script and package.json rewrite script, which have been removed. - Updated the `exports` field in `package.json` to provide proper entrypoints and type definitions for both browser and Node.js consumers, ensuring correct module resolution and compatibility with modern bundlers. **Source code refactoring & browser/node separation:** - Moved browser-specific connection classes to a dedicated `src/browser/` directory, renamed classes for clarity (e.g., `WebrtcConnection` instead of `BrowserWebrtcConnection`), and updated imports throughout the codebase to use consistent paths and new module aliases. - Introduced browser stubs for Node.js-only features (e.g., `WebsocketServer`, `GeoIpLocator`, `AutoCertifierClient`), which throw errors when used in the browser, improving developer experience and preventing runtime failures. **Testing & module resolution:** - Added and updated module aliasing in test and build configs (`jest.config.ts`, `karma.config.ts`) to ensure the correct environment-specific implementations are used during testing and bundling. **Code quality & internal refactors:** - Improved code clarity by updating imports to use named imports for `EventEmitter`, moving utility functions to dedicated helper files, and cleaning up legacy or redundant code. **Documentation:** - Updated documentation to reflect refactored class names, improving consistency between code and docs.
github-actions
bot
added
network
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.