Add language neutral protocol conformance fixtures for concore core behavior

[Titas-Ghosh]

Hi @pradeeban ,
I’ve been looking through the recent CI and testing improvements (like #196, #193, and #210), and the test setup is looking really solid but then it got me thinking about a follow up improvement that could make maintaining the project much easier in the long run.

As the project evolves, there's always a risk that protocol behavior might drift slightly across different implementations. Right now, we don’t really have a single, central "source of truth" that defines exactly how the protocol must behave, regardless of the language it's written in.

what i am proposing is a Language Neutral Conformance Suite ,which is basically adding a fixture based conformance layer. Essentially, we would create a set of standardized test cases (fixtures) that make the expected behavior explicit and easy to validate across the board.

What this would look like:

• Protocol Fixtures: A collection of JSON input/expected output pairs covering core behaviors (e.g., initval handling, read/write framing semantics, param parsing edge cases, and fallback behaviors for malformed inputs).
• A pytest Conformance Runner: A tool to validate our current Python behavior against these new fixtures.
• Implementation Agnostic Format: Keeping the test vectors completely language neutral so that other runtimes can reuse the exact same files later to prove compatibility.

Why i think it would be a very beneficial thing to add because it creates :

• Zero Ambiguity: It creates a crystal clear baseline for how the protocol should act.
• Future-Proofing: Gives us a strict compatibility standard for any new runtime implementations we build down the line.
• Safety: Catches weird edge-case regressions early.

If this sounds like a good direction to you, I’d be happy to open a focused PR to get it started.

For the first iteration, i would just introduce the fixture schema, add an initial set of tests, and wire it into our existing pytest runs. There would be absolutely no runtime behavior changes in this first step, i will be just laying the groundwork.

Let me know what you think! I'm happy to pivot if you have a different vision for the testing architecture.

[pradeeban]

@Titas-Ghosh I like the idea. I just fear the practical challenges. Language bindings were contributed by different folks. There could be reasons for the way things diverging across languages. Making changes to change the conformance could break certain implementations (say, Verilog.) How do you plan to implement this without incurring such issues?

[Titas-Ghosh]

Hi @pradeeban , what you have said is a completely valid concern which even i thought of and i totally agree, especially with bindings like Verilog that have very different architectural needs, forcing them into a rigid box could cause major headaches.

My goal isn't to force all languages to blindly mimic Python. Instead, I want to use this to map out what the protocol behavior actually is across the board. To ensure we don't break existing implementations, I'd like to propose a strict "observe first, enforce later" approach.

Here is what i have thought of planning out to avoid even minimum risks:

Phase 1: Foundation & Observation (Zero Risk)

We establish the test structure without changing any runtime code or failing any builds.

• PR 1: The Blueprint & Python Baseline
  • What it does: Introduces the language neutral JSON fixture schema (e.g., tests/protocol_fixtures/*.json) and a simple conformance runner for Python.
  • Safety check: In CI, this will run in a "report-only" mode (continue-on-error). It will not fail any existing builds; it will simply publish a report of how Python behaves against the expected standard.

Phase 2: Mapping the Territory

We expand the tests to other bindings(no forced compliance) just to document reality.

• PR 2: The Cross Language Audit
  • What it does: Adds test runners/adapters for the other language bindings (where practical).
  • Safety check: When we find a mismatch between the binding and the JSON fixture, we don't automatically fix the binding. Instead, we review it together and classify it:
    • required (it's a genuine bug we all agree needs fixing)
    • implementation_defined (a completely valid, language-specific design choice)
    • known_deviation (a quirk we decide to document and leave alone)

Phase 3: Controlled Hardening (Opt-in)

We only touch runtime code if we explicitly agreed it was necessary in Phase 2.
This phase will only be done after review/approval of Phase 2

• PR 3+: Targeted, Single Binding Fixes
  • What it does: Addresses only the deviations we classified as required bugs.
  • Safety check: We do this one binding at a time, in very small PRs. No massive cross language refactoring. if a change threatens backward compatibility, we simply revert it to a known_deviation.

I think this phased approach is the best way to go to for safely implementing this, i’d be happy to just draft PR 1 so you can see what the JSON schema looks like in practice. If it feels too risky once you see the code, we can easily pause or scrap it.

How does it sound? I happy to iterate based on your suggestions .
Again thanks a lot for the feedback

[pradeeban]

@Titas-Ghosh ok, go ahead. Having a PR will help me understand the implications better.

[Titas-Ghosh]

Thanks a lot @pradeeban!
I’ll get started on PR 1 right away. i’ll keep it strictly focused on the JSON schema and the Python runner in report only mode. That way, you can see exactly how the infrastructure looks and feels without any risk to the existing builds.
I'll ping you as soon as it's up for review!

[pradeeban]

Solid plan, @Titas-Ghosh.

[Titas-Ghosh]

Hi @pradeeban, thanks a lot again for approving my plan!
I have just opened the Phase 1 PR here: #443
As promised, it is strictly the baseline infrastructure and the Python runner in report only mode, i have not touched any runtime code. Let me know if this structure makes sense to you once you have a chance to take a look!