feat: json, json-all generators #287

flakey5 · 2025-05-31T18:21:03Z

Closes #214

TODO:

Dedupe code for type list parsing (re feat: json, json-all generators #287 (comment))

codecov-commenter · 2025-05-31T18:21:34Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
346	2	344	0

View the top 2 failed test(s) by shortest run time

test::module.md

Stack Traces | 0.0146s run time

[Error [ERR_TEST_FAILURE]: The expression evaluated to a falsy value:

  assert.ok(validator.validate(result[0], schema).valid)
] {
  code: 'ERR_TEST_FAILURE',
  failureType: 'testCodeFailure',
  cause: AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
  
    assert.ok(validator.validate(result[0], schema).valid)
  
      at TestContext.<anonymous> (file:.../json/__tests__/index.test.mjs:45:14)
      at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
      at async Test.run (node:internal/test_runner/test:1113:7)
      at async Suite.processPendingSubtests (node:internal/test_runner/test:788:7) {
    generatedMessage: true,
    code: 'ERR_ASSERTION',
    actual: false,
    expected: true,
    operator: '==',
    diff: 'simple'
  }
}

test::text-doc.md

Stack Traces | 0.0596s run time

Error [ERR_TEST_FAILURE]: The expression evaluated to a falsy value:

  assert.ok(validator.validate(result[0], schema).valid)

    at async Promise.all (index 0) {
  code: 'ERR_TEST_FAILURE',
  failureType: 'testCodeFailure',
  cause: AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
  
    assert.ok(validator.validate(result[0], schema).valid)
  
      at TestContext.<anonymous> (file:.../json/__tests__/index.test.mjs:45:14)
      at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
      at async Test.run (node:internal/test_runner/test:1113:7)
      at async Promise.all (index 0)
      at async Suite.run (node:internal/test_runner/test:1518:7)
      at async startSubtestAfterBootstrap (node:internal/test_runner/harness:358:3) {
    generatedMessage: true,
    code: 'ERR_ASSERTION',
    actual: false,
    expected: true,
    operator: '==',
    diff: 'simple'
  }
}

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

avivkeller · 2025-09-30T01:14:13Z

@flakey5 is this still in progress? Need any help?

flakey5 · 2025-09-30T01:54:04Z

Still in progress just haven't been committing, I do wanna get this ready for review within the next coming weeks though

vercel · 2025-10-06T20:16:35Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Review	Updated (UTC)
api-docs-tooling	Ready	Preview	Jan 8, 2026 5:24am

flakey5 · 2025-11-10T16:47:17Z

Bulk of this is done, with only two main things left:

Couple of failing tests for createParameterGroupings
Adding the @constructor property to class sections

I'm leaving this as a draft until those are done, but the rest is ready for review

avivkeller

the rest is ready for review

I've left a first round of reviews. Thank you so much for this effort!

src/utils/generator-error.mjs

src/generators/json/utils/stringifyNode.mjs

src/generators/json/utils/createSectionBase.mjs

src/generators/json/utils/createMethodSection.mjs

src/generators/json/utils/createModuleSection.mjs

src/generators/json/constants.mjs

src/generators/json-all/util/generateJsonSchema.mjs

src/generators/json-all/index.mjs

src/generators/index.mjs

tools/generate-json-types.mjs

flakey5 · 2025-11-29T02:49:36Z

Marking ready for review, should be all there minus #287 (comment)

src/generators/json-all/index.mjs

avivkeller · 2025-11-29T18:28:59Z

src/generators/json-all/index.mjs

+      switch (section.type) {
+        case 'module':
+          generatedValue.modules.push(copiedSection);
+          break;
+        case 'text':
+          generatedValue.text.push(copiedSection);
+          break;
+        default:
+          throw new TypeError(`unsupported root section type ${section.type}`);
+      }


Suggested change

switch (section.type) {

case 'module':

generatedValue.modules.push(copiedSection);

break;

case 'text':

generatedValue.text.push(copiedSection);

break;

default:

throw new TypeError(`unsupported root section type ${section.type}`);

}

generatedValue[type === 'module' ? 'modules' : type].push(copiedSection)

nit

-1 imo, even though the error shouldn't ever be thrown I don't think we should assume it won't be. Otherwise if we do get a section that's not a module or text in the input, we run the risk of violating the json schema and not knowing about it

src/generators/json-all/index.mjs

src/generators/json-all/util/generateJsonSchema.mjs

src/generators/json/constants.mjs

src/generators/json/index.mjs

src/generators/json/utils/sections/index.mjs

src/utils/generator-error.mjs

ovflowd · 2025-11-30T16:17:03Z

src/generators/json/utils/sections/event.mjs

+          );
+        }
+
+        if (rest[0]?.type === 'link') {


Can we have these specific extractors on dedicated function for ... extracting these? Also are the types here only link and "else"?

src/generators/json/generated/generated.d.ts

src/generators/json/index.mjs

ovflowd · 2025-11-30T17:17:32Z

src/generators/json/schema.jsonc

Shouldn't this be automatically generated?

The schema should be maintained by hand. It doesn't need to be in a standalone json file though, we could have it be a function like with the json-all's schema. Only reason it is standalone currently is to make it so we can have better intellisense/autocomplete in editors when modifying the schema

scripts/generate-json-types.mjs

ovflowd

Mostly LGTM, I think that we can dedup lots of the code, extract many of the utility functions within parsing things (such as event, property) into dedicated tiny functions that do one thing and then unit test them.

Makes the parent functions simpler, easier to read the code and understand what pieces does what, reusability and unit testing.

avivkeller · 2025-12-03T22:51:31Z

@flakey5 Is there a reason why many properties start with @? I feel like that makes it harder for consumers, since they now need to <>.["@prop"] instead of just <>.prop

Copilot

Pull request overview

This PR introduces new JSON generators (json and json-all) for the Node.js documentation toolkit, providing machine-readable JSON representations of API documentation.

Key Changes:

New json generator that outputs individual JSON files for each API module with comprehensive type information, method signatures, properties, and events
New json-all generator that combines all JSON outputs into a single file
Comprehensive JSON schema definition with TypeScript type generation support

Reviewed changes

Copilot reviewed 39 out of 44 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
`src/generators/json/index.mjs`	Main JSON generator implementation
`src/generators/json-all/index.mjs`	Aggregator generator for combined JSON output
`src/generators/json/schema.jsonc`	JSON schema definition for documentation structure
`src/generators/json/utils/sections/*.mjs`	Section parsers for modules, classes, methods, properties, and events
`src/generators/json/utils/parseTypeList.mjs`	Type list parsing utility
`src/generators/json/utils/createParameterGroupings.mjs`	Method signature parameter grouping logic
`src/utils/buildHierarchy.mjs`	Shared utility for building hierarchical entry structures
`src/utils/assertAstType.mjs`	AST node type assertion utilities
`src/utils/generator-error.mjs`	Custom error class for generator errors
`src/utils/unist.mjs`	Enhanced node transformation with support for break, delete, and link nodes
`scripts/generate-json-types.mjs`	Script to generate TypeScript definitions from JSON schema
`package.json`	New dependencies for JSON parsing and schema validation
`.github/workflows/generate.yml`	CI workflow integration for JSON generator

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/generate-json-types.mjs

src/generators/json/utils/sections/method.mjs

src/generators/json/utils/sections/__tests__/base.test.mjs

src/generators/json/schema.jsonc

Copilot · 2025-12-03T22:56:14Z

src/generators/json/schema.jsonc

+          "$ref": "#/definitions/MethodReturnType",
+        },
+      },
+      "required": ["parameters", "@returns"],


[nitpick] The MethodSignature.parameters property is defined as required (line 233), but in several places in the code, signatures are created with an empty parameters array instead of omitting it when there are no parameters. Consider making this property optional or consistently handling empty arrays.

Suggested change

"required": ["parameters", "@returns"],

"required": ["@returns"],

src/generators/json/utils/sections/index.mjs

src/generators/json/schema.jsonc

avivkeller

More comments :-)

avivkeller · 2025-12-03T22:47:41Z

src/generators/json/utils/sections/base.mjs

+
+    // Not code, let's stringify it and add it to the description.
+    section.description ??= '';
+    section.description += `${transformNodeToString(node)}${node.type === 'paragraph' ? '\n' : ' '}`;


Is this added newline needed?

src/generators/json/utils/sections/base.mjs

avivkeller · 2025-12-03T22:50:17Z

src/generators/json/utils/sections/base.mjs

+      } else {
+        section['@example'] = node.value;
+      }
+


Suggested change

const examples = [...enforceArray(section['@example']), node.value];

section['@example'] = examples.length === 1 ? examples[0] : examples;

src/generators/json/utils/sections/base.mjs

src/generators/json/utils/sections/index.mjs

avivkeller · 2025-12-04T21:17:52Z

src/generators/json/utils/sections/index.mjs

+  if (section.type !== 'module' && section.type !== 'text') {
+    throw new GeneratorError(
+      `expected root section to be a module or text, got ${section.type}`,
+      { entry: head }
+    );
+  }
+


I think these if checks are too strict, IMO. I feel like there's nothing wrong with it having the wrong type (like, the generator should still work)

The generator would work, but it would be violating the json schema which consumers would expect to be the source of truth

src/generators/json/constants.mjs

src/generators/json-all/index.mjs

src/generators/json/utils/findParentSection.mjs

Signed-off-by: flakey5 <73616808+flakey5@users.noreply.github.com>

flakey5 · 2026-01-08T01:58:46Z

src/generators/json/utils/sections/index.mjs

+
+  return {
+    $schema: `${BASE_URL}docs/${version}/api/${SCHEMA_FILENAME}`,
+    source: head.api_doc_source,


Currently this is something like doc/api/buffer.md, I wonder if it'd be more useful to give an actual link to the source file like https://nodejs.org/docs/v25.0.0/api/buffer.md

Signed-off-by: flakey5 <73616808+flakey5@users.noreply.github.com>

Co-authored-by: Aviv Keller <me@aviv.sh>

Signed-off-by: flakey5 <73616808+flakey5@users.noreply.github.com>

avivkeller mentioned this pull request Jun 24, 2025

feat(web): add web generator #285

Merged

6 tasks

flakey5 force-pushed the flakey5/20240909/new-json-gen branch from a4cd425 to d711fee Compare October 6, 2025 20:16

vercel bot deployed to Preview October 6, 2025 20:19 View deployment

vercel bot deployed to Preview October 6, 2025 23:19 View deployment

flakey5 mentioned this pull request Oct 11, 2025

Making it easier for DTS consumers #441

Open

flakey5 force-pushed the flakey5/20240909/new-json-gen branch from 2d10df1 to e4dbbe4 Compare October 23, 2025 04:17

vercel bot deployed to Preview October 23, 2025 04:20 View deployment

vercel bot deployed to Preview October 23, 2025 18:16 View deployment

vercel bot deployed to Preview October 27, 2025 00:44 View deployment

flakey5 force-pushed the flakey5/20240909/new-json-gen branch from 6bfc555 to 1bcf93d Compare November 10, 2025 16:42

vercel bot deployed to Preview November 10, 2025 16:45 View deployment

avivkeller reviewed Nov 12, 2025

View reviewed changes

vercel bot deployed to Preview November 16, 2025 08:02 View deployment

vercel bot deployed to Preview November 29, 2025 01:15 View deployment

flakey5 force-pushed the flakey5/20240909/new-json-gen branch from ee1684d to b37daee Compare November 29, 2025 02:34

vercel bot deployed to Preview November 29, 2025 02:36 View deployment

flakey5 force-pushed the flakey5/20240909/new-json-gen branch from b37daee to 8ebb3d6 Compare November 29, 2025 02:44

vercel bot deployed to Preview November 29, 2025 02:46 View deployment

flakey5 marked this pull request as ready for review November 29, 2025 02:49

flakey5 requested a review from a team as a code owner November 29, 2025 02:49

avivkeller reviewed Nov 29, 2025

View reviewed changes

vercel bot deployed to Preview November 30, 2025 05:19 View deployment

vercel bot deployed to Preview November 30, 2025 06:33 View deployment

ovflowd reviewed Nov 30, 2025

View reviewed changes

src/generators/json/generated/generated.d.ts Show resolved Hide resolved

ovflowd reviewed Nov 30, 2025

View reviewed changes

src/generators/json/index.mjs Outdated Show resolved Hide resolved

ovflowd reviewed Nov 30, 2025

View reviewed changes

scripts/generate-json-types.mjs Show resolved Hide resolved

ovflowd reviewed Nov 30, 2025

View reviewed changes

vercel bot deployed to Preview December 1, 2025 01:38 View deployment

avivkeller requested a review from Copilot December 3, 2025 22:52

Copilot started reviewing on behalf of avivkeller December 3, 2025 22:53 View session

Copilot finished reviewing on behalf of avivkeller December 3, 2025 22:54

Copilot AI reviewed Dec 3, 2025

View reviewed changes

avivkeller added this to the Web Generator Migration milestone Dec 3, 2025

avivkeller reviewed Dec 4, 2025

View reviewed changes

avivkeller mentioned this pull request Dec 23, 2025

Intertwined JavaScript + Markdown Parsing #530

Open

feat: json, json-all generators

0ec318d

Signed-off-by: flakey5 <73616808+flakey5@users.noreply.github.com>

flakey5 force-pushed the flakey5/20240909/new-json-gen branch from 1cf5261 to 0ec318d Compare January 7, 2026 23:54

vercel bot had a problem deploying to Preview January 7, 2026 23:54 Failure

flakey5 commented Jan 8, 2026

View reviewed changes

review, some things broken

b5d67be

Signed-off-by: flakey5 <73616808+flakey5@users.noreply.github.com>

vercel bot had a problem deploying to Preview January 8, 2026 02:03 Failure

Apply suggestions from code review

db2ce37

Co-authored-by: Aviv Keller <me@aviv.sh>

vercel bot had a problem deploying to Preview January 8, 2026 02:05 Failure

(hopefully) fix ci

fc0bde9

Signed-off-by: flakey5 <73616808+flakey5@users.noreply.github.com>

vercel bot had a problem deploying to Preview January 8, 2026 02:30 Failure

actual fix ci

e684db8

Signed-off-by: flakey5 <73616808+flakey5@users.noreply.github.com>

vercel bot deployed to Preview January 8, 2026 03:45 View deployment

actually fix ci

557b9b6

Signed-off-by: flakey5 <73616808+flakey5@users.noreply.github.com>

vercel bot deployed to Preview January 8, 2026 04:56 View deployment

little bit more coverage for methods

ae34640

Signed-off-by: flakey5 <73616808+flakey5@users.noreply.github.com>

vercel bot deployed to Preview January 8, 2026 05:24 View deployment

	"required": ["parameters", "@returns"],
	"required": ["@returns"],


	const examples = [...enforceArray(section['@example']), node.value];
	section['@example'] = examples.length === 1 ? examples[0] : examples;

feat: json, json-all generators #287

Are you sure you want to change the base?

feat: json, json-all generators #287

Uh oh!

Conversation

flakey5 commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented May 31, 2025 • edited by codecov bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

avivkeller commented Sep 30, 2025

Uh oh!

flakey5 commented Sep 30, 2025

Uh oh!

vercel bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flakey5 commented Nov 10, 2025

Uh oh!

avivkeller left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

flakey5 commented Nov 29, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ovflowd Nov 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ovflowd left a comment

Choose a reason for hiding this comment

Uh oh!

avivkeller commented Dec 3, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

flakey5 commented May 31, 2025 •

edited

Loading

codecov-commenter commented May 31, 2025 •

edited by codecov bot

Loading

vercel bot commented Oct 6, 2025 •

edited

Loading

ovflowd Nov 30, 2025 •

edited

Loading