Generic Source Provenance #2099

joshua-zivkovic · 2026-01-07T08:29:08Z

Towards #2069

This PR changes the approach to how buildstream handles source provenance information, moving from a hard coded implementation of attributes (which buildstream core needs to know nothing about) to a more generic, user defined approach where buildstream only needs to carry the values to the next step.

Users can now define their projects allowed set of source provenance attributes using the source_provenance_fields key in the project.conf. Where attributes are used in source provenance but not defined in the project.conf, buildstream will refuse attribute and throw an error informing the user that the attribute is not present in the project.conf

Currently there is no limitation on these values since neither plugins nor buildstream sbom are yet using this approach so there is no defined set of attributes plugins and users should refer to but this will soon be added to buildstream-sbom as it will depend on what buildstream-sbom supports for SPDX generation.

Note: This PR previously included updates to source track to allow plugins to automatically populate these source attributes where possible; this has been moved to a follow up PR.

juergbi · 2026-01-09T10:15:08Z

src/buildstream/source.py

+    def load_source_provenance(self, node: MappingNode) -> None:
+        raise ImplError("Source plugin '{}' does not implement load_source_provenance()".format(self.get_kind()))
+
+    def get_source_provenance(self) -> SourceProvenance:
+        raise ImplError("Source plugin '{}' does not implement get_source_provenance()".format(self.get_kind()))
+
+    def set_source_provenance(self, source_provenance: SourceProvenance):
+        raise ImplError("Source plugin '{}' does not implement set_source_provenance()".format(self.get_kind()))


To retain backward compatibility, we can't require source plugins to implement these methods. I.e., we need (trivial) default implementations for any method that is called unconditionally by the core. API documentation will be required as well, of course.

That's easy enough if we just go with a "do nothing" implementation

juergbi · 2026-01-09T10:17:32Z

src/buildstream/source.py

+    def set_source_provenance(self, source_provenance: SourceProvenance):
+        raise ImplError("Source plugin '{}' does not implement set_source_provenance()".format(self.get_kind()))
+
+    def track(self, *, previous_sources_dir: Optional[str] = None) -> (SourceRef, SourceProvenance):


We need to find a way to allow returning a SourceProvenance without breaking API.

Hmm, that might be more difficult to work out. I'll see if I can work something sensible out

For now I've gone with combining the old and new return signatures (https://github.com/apache/buildstream/pull/2099/files#diff-48200ef302e5d841cce81834e8879a65e59db922a22605b9d7fa16b4dd5c7067R941) and added some logic to unpackage it appropriately (https://github.com/apache/buildstream/pull/2099/files#diff-48200ef302e5d841cce81834e8879a65e59db922a22605b9d7fa16b4dd5c7067R1811). I believe that's all that's required to not break the API here

juergbi · 2026-01-16T10:27:33Z

src/buildstream/element.py

+                        project._project_conf.get_mapping("source-provenance-fields", None)
+                        or project.source_provenance_fields


Is this the logic to handle the default on a project level? I'm not sure whether this works correctly, however, this logic should in any case be in the Project class such that Element and Source can simply use project.source_provenance_fields (_project.conf is considered private and should not be accessed from other classes). Or am I missing a reason why this can't be done in the Project class?

Thanks, I've made the change to move the logic to Project instead

juergbi · 2026-01-16T10:29:05Z

Have you already started looking into adding test(s) for this?

juergbi · 2026-01-16T10:35:32Z

src/buildstream/source.py

+                )
+
+            # Make sure everything is a string
+            provenance = MappingNode.from_dict({key: value.as_str() for key, value in provenance_node.items()})


Is there a reason we actually need to use a newly created MappingNode instead of directly using provenance_node (after verification)? I suspect that this recreation loses provenance of the provenance node (i.e., .bst file name and line number).

I wasn't aware I could simply write back to a MappingNode like a dictionary, even though the docs clearly state that. Regardless, this is no longer being used since this logic has been changed to throwing an error in the event that the value could not be parsed as a string

juergbi · 2026-01-16T10:37:15Z

src/buildstream/source.py

-        self.homepage: Optional[str] = homepage
-        """
-        The project homepage URL
-        """
-
-        self.issue_tracker: Optional[str] = issue_tracker


These attributes are public, we can't drop them. We need to populate them from generic source provenance for backwards compatibility.

Sorry, I keep messing up the backwards compatibility. I've added these back in

juergbi

Somewhat of a nit but it seems we're mixing two terms for the same thing. I'm not sure whether "source provenance field" or "source provenance attribute" is a better fit. However, whatever we choose, we should be consistent to avoid confusion. Or is there a semantic distinction that I'm overlooking?

I think I used the term 'field' first because I thought the same term was used in the documentation already, but I may be mistaken. I'm not dead-set on the 'field' term. If 'attribute' is a better fit with the rest of BuildStream's API and documentation, I'm happy to settle on that as long as we're consistent.

src/buildstream/element.py

joshua-zivkovic · 2026-01-23T10:32:53Z

Somewhat of a nit but it seems we're mixing two terms for the same thing. I'm not sure whether "source provenance field" or "source provenance attribute" is a better fit. However, whatever we choose, we should be consistent to avoid confusion. Or is there a semantic distinction that I'm overlooking?

I've noticed this too and haven't got a fixed opinion on which we use. I was just using "attributes" since #2069 uses the term. I'm not seeing any deciding usage in the docs.

In context of errors that already exist, it seems "attribute" reads better than "field", to me at least, but that's just how I've been reading it all this time anyway.

I'll change up the error messages as per your suggestions and make the term used consistent based on what reads best from there

joshua-zivkovic · 2026-01-23T15:15:55Z

I've seemingly messed something up with the source provenance attribute tests so they're currently failing. I'll look at this next week

This allows multi-source source plugins to provide this information per source rather than as a singular top level. This is done by adding a `provenance_node` parameter to `create_source_info()` that when specified overrides the use of the source's top level source provenance info. Co-authored-by: Joshua Zivkovic <joshua.zivkovic@codethink.co.uk>

Remove harcoded SPDX attributes and make them be generic instead. Project allowed attributes are configured via the project config, these supported values a determined by buildstream-sbom's support Co-authored-by: Jürg Billeter <j@bitron.ch> Co-authored-by: Joshua Zivkovic <joshuazivkovic@codethink.co.uk>

src/buildstream/source.py

joshua-zivkovic force-pushed the JZ/source-provenance branch 3 times, most recently from d18f672 to 6894c4f Compare January 8, 2026 14:13

juergbi reviewed Jan 9, 2026

View reviewed changes

joshua-zivkovic force-pushed the JZ/source-provenance branch 6 times, most recently from 51ff651 to 98735d0 Compare January 14, 2026 14:15

joshua-zivkovic marked this pull request as ready for review January 14, 2026 14:18

joshua-zivkovic requested review from BenjaminSchubert, abderrahim, cs-shadow and gtristan as code owners January 14, 2026 14:18

joshua-zivkovic force-pushed the JZ/source-provenance branch 5 times, most recently from 182c33a to 74c718a Compare January 16, 2026 10:21

juergbi reviewed Jan 16, 2026

View reviewed changes

joshua-zivkovic force-pushed the JZ/source-provenance branch 5 times, most recently from df0f7d9 to a91adf9 Compare January 20, 2026 07:50

joshua-zivkovic force-pushed the JZ/source-provenance branch 2 times, most recently from 3f4f475 to 34dd8b0 Compare January 21, 2026 11:16

juergbi reviewed Jan 23, 2026

View reviewed changes

src/buildstream/element.py Outdated Show resolved Hide resolved

src/buildstream/element.py Outdated Show resolved Hide resolved

joshua-zivkovic force-pushed the JZ/source-provenance branch from 34dd8b0 to 1ea1f4d Compare January 23, 2026 15:15

joshua-zivkovic force-pushed the JZ/source-provenance branch from 1ea1f4d to a13a331 Compare January 26, 2026 08:01

juergbi and others added 5 commits January 26, 2026 08:06

tests/frontend/show.py: Add source info test for multi-source plugin

0cb0e5e

Add source provenance attribute tests

8409e5d

Add source provenance attribute tests

749602a

joshua-zivkovic force-pushed the JZ/source-provenance branch from a13a331 to 749602a Compare January 26, 2026 08:12

franred reviewed Jan 26, 2026

View reviewed changes

src/buildstream/source.py Outdated Show resolved Hide resolved

		project._project_conf.get_mapping("source-provenance-fields", None)
		or project.source_provenance_fields

Generic Source Provenance #2099

Are you sure you want to change the base?

Generic Source Provenance #2099

Uh oh!

Conversation

joshua-zivkovic commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juergbi commented Jan 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juergbi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

joshua-zivkovic commented Jan 23, 2026

Uh oh!

joshua-zivkovic commented Jan 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joshua-zivkovic commented Jan 7, 2026 •

edited

Loading