diff --git a/Standards/scs-0003-v1-sovereign-cloud-standards-yaml.md b/Standards/scs-0003-v1-sovereign-cloud-standards-yaml.md index 3e5491793..b9527f0a2 100644 --- a/Standards/scs-0003-v1-sovereign-cloud-standards-yaml.md +++ b/Standards/scs-0003-v1-sovereign-cloud-standards-yaml.md @@ -86,22 +86,23 @@ While instructive, this view is still a bit simplified. Let's get more precise n A statement about the subject that can be evaluated unambiguously to be either satisfied or not. The result is either `PASS` or `FAIL`, or—if the test could not be performed—`DNF` (did not finish). A test case can be as simple as "the subject conforms to standard X", but a standard can also be decomposed into multiple test cases, which can then be reported on (also to the customers) individually. This latter option has the advantage that we can show explicitly if the subject complies with optional parts of the standard. -3. _Check_: - A script that determines and reports the results of certain test cases. The report is printed to stdout, and each test case is reported as a single line of the form `testcase-id: [PASS/FAIL]`. The result `DNF` is not reported. Lines of other forms are permissible and will be ignored. - We also occasionally extend the concept of _check_ to manual audits. -4. _Module_: - A collection of test cases and corresponding checks, together with additional meta information such as the result lifetime, description, and a list of tags for a test case. - Ultimately, we aim to specify one module for each version of each standard: the module translates the standard into something measurable and, ideally, executable to be used for certification. -5. _Selector (expression)_: - An expression used to select test cases by referring to the tags that must (or must not) be present. -6. _Target_: - A named collection of test cases specified using selector expressions. +3. _Script_: + A script that determines and reports the results of certain test cases, + which are specified with the script, together with additional meta information such as the result lifetime, and a description for each test case. + The report is printed to stdout, and each test case is reported as a single line of the form `testcase-id: [PASS/FAIL/ABORT]`. + Lines of other forms are permissible and will be ignored. + We also occasionally extend the concept of _script_ to manual audits. +4. _Target_: + A named collection of test cases. Ultimately, the certification of a subject always depends on a designated "main" target; all its test cases must be passed for the certificate to be awarded. Further targets can be used to report on optional aspects of the certificate, such as particularly good security and encryption measures. -7. _(Certificate-scope) version_: - A collection of modules and a collection of targets, one of them being "main". +5. _Module_: + A collection of targets. + Ultimately, we aim to specify one module for each version of each standard: the module translates the standard into something measurable and, ideally, executable to be used for certification. +6. _(Certificate-scope) version_: + A collection of modules. Like a module, the version has targets, but they are implicitly given by taking the union of the targets of all its modules. Note that a collection of modules can again be construed as a (larger) module. We opt to use one module per standard version, as mentioned above, in order to make commonalities between certificate-scope versions explicit. -8. _Certificate scope_: +7. _Certificate scope_: A list of certificate-scope versions. Having introduced these concepts, we can now get even more precise by defining the actual specification in YAML format. @@ -120,6 +121,7 @@ The certification YAML _MUST_ contain the following keys: | `uuid` | String | Universally unique identifier | `d912d0a5-826a-4b01-bafd-b48f65f76f43` | | `name` | String | Full name of this certificate scope | `SCS-open KaaS` | | `url` | String | Valid URL to the latest raw version of this document | `https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/scs-open-kaas.yaml` | +| `scripts` | Array of maps | List of script descriptors (described below) | (see below) | | `modules` | Array of maps | List of module descriptors (described below) | (see below) | | `timeline` | Array of maps | List of timeline entries (described below) | (see below) | | `versions` | Array of maps | List of version descriptors (described below) | (see below) | @@ -154,7 +156,6 @@ then a certificate of that prerequisite scope has to be presented before the cer | --------------- | ------------- | ------------------------------------------------------------------------------- | ------------------ | | `version` | String | required: version of the particular list of standards | `v3` | | `include` | Array | required: list of module ids or include descriptors (see below) | `[scs-0100-v3]` | -| `targets` | Map of maps | required: this maps target names to selector expressions (explained below) | `main: mandatory` | | `stabilized_at` | Date | ISO formatted date indicating the date after this version is considered stable. | `2022-11-09` | The ids of the test cases of all the modules specified via `include` MUST be pairwise different. @@ -172,38 +173,6 @@ Each include may be specified by means of a module id (i.e., a string) or by an When the referenced module uses parameters, then these parameters must be assigned values here. -#### Selector expressions - -In order to define what a selector expression is, we need to define tags, atoms and terms first. - -A _tag_ is a string that does not contain any space, comma, forward slash, or exclamation mark. - -Examples: `iaas`, `mandatory`, `recommended`, `encryption`. - -An _atom_ is a string that is either (i) a tag or (ii) an exclamation mark followed by tag. -A list of tags _satisfies_ the atom if - -- the atom is of form (i) and the tag is contained in the list, or -- the atom is of form (ii) and the tag is not contained in the list. - -Examples: `mandatory`, `!mandatory`. - -A _term_ is a string that is a non-empty list of atoms joined by slashes. -A list of tags _satisfies_ the term if it satisfies at least one of the atoms. - -Examples: `mandatory`, `mandatory/recommended`, `!mandatory/encryption`. - -A _selector (expression)_ is a string that is a non-empty list of terms joined by space. -A list of tags _satisfies_ the selector if it satisfies all the terms. - -Examples: `mandatory`, `iaas mandatory`, `iaas !mandatory/encryption`. - -In the map `targets` above, it is possible to specify a list of selectors that are joined by comma. -(Note that this is still a string, not a YAML list.) -A list of tags satisfies this list of selectors if it satisfies at least one of the selectors. - -Examples: `mandatory iaas, recommended kaas` (NOT: `[mandatory iaas, recommended kaas]`) - ### Module descriptor | Key | Type | Description | Example | @@ -212,8 +181,7 @@ Examples: `mandatory iaas, recommended kaas` (NOT: `[mandatory iaas, recommended | `name` | String | name of this module | `Flavor naming v3` | | `url` | String | Valid URL to relevant documentation (usually a standard document) | `https://docs.scs.community/standards/scs-0100-v3-flavor-naming` | | `parameters` | List | List of parameters that the checks in this module might use | `[image_spec]` | -| `run` | Array | List of all checks that should be run; each entry being a check descriptor | (see below) | -| `testcases` | Array | List of all test cases; each entry being a test-case descriptor | (see below) | +| `targets` | Map of lists | required: this maps target names to lists of testcases | `main: [tc1, tc2]` | The parameters specified here will be added to the variable assignment for all check tools that belong to this module, so they will be substituted in the same way. The values to these parameters must be provided in the include descriptor as explained above. @@ -223,13 +191,14 @@ Using parameters offers two advantages: - they may show up in the automatically generated documentation, whereas the check tools themselves probably won't. - multiple versions of a standard can be represented using the same module, if everything that changes between versions can be captured by the parameters. -### Check descriptor +### Script descriptor -The following fields are valid for every check descriptor: +The following fields are valid for every script descriptor: | Key | Type | Description | Example | | ----------------- | ------ | ---------------------------------------------------------------------------------------------- | ---------- | | `section` | String | _Optional_ what section to associate this check with (sections can be checked in isolation) | `weekly` | +| `testcases` | Array | List of all test cases; each entry being a test-case descriptor | (see below) | Additional fields are valid depending on whether the check is automated or manual. @@ -257,14 +226,11 @@ TBD | ----------------- | --------------- | ------------------------------------------------------------------------------------------------- | ----------------- | | `id` | String | Identifier for this test case (immutable and unique within this module) | `image-md-check` | | `lifetime` | String | One of: `day`, `week` (_default_), `month`, `quarter`, `year` | `day` | -| `tags` | List of strings | A tag is a keyword that will be used to select this test case using a selector expression | `[mandatory]` | | `description` | String | Short description of the test case | | A test result is valid until the end of the next period, except when lifetime is `year`: then the result is valid until the end of the following month plus one year. -A tag MUST NOT contain any of these characters: space, comma, exclamation mark, forward slash. - The `id` of a test case MUST NOT be changed. Exceptions MAY be made if the test case is not referenced by any stable version. diff --git a/Tests/scs-compatible-iaas.yaml b/Tests/scs-compatible-iaas.yaml index 3513e5e92..3bc5fd0fc 100644 --- a/Tests/scs-compatible-iaas.yaml +++ b/Tests/scs-compatible-iaas.yaml @@ -7,6 +7,195 @@ uuid: 50393e6f-2ae1-4c5c-a62c-3b75f2abef3f url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/scs-compatible-iaas.yaml variables: - os_cloud +scripts: +- executable: ./iaas/openstack_test.py + args: -c {os_cloud} {testcases} + testcases: + - id: scs-0100-syntax-check + description: Flavor names starting `SCS-` comply with syntax. + - id: scs-0100-semantics-check + description: Syntactically correct SCS flavor names represent the truth. + - id: scs-0101-flavor-property + description: Must have all flavor properties recommended in + - id: scs-0101-image-property + description: Must have all image properties recommended in + - id: scs-0101-rngd + description: Images of the test sample must have the service `rngd`; see + - id: scs-0101-entropy-avail + description: A test instance must have the correct `entropy_avail`; see + - id: scs-0101-fips-test + description: A test instance must pass the "FIPS test"; see + - id: scs-0102-prop-architecture + description: Each image has a meaningful value for `architecture`. + - id: scs-0102-prop-hash_algo + description: Each image has a meaningful value for `hash_algo`. + - id: scs-0102-prop-min_disk + description: Each image has a meaningful value for `min_disk`. + - id: scs-0102-prop-min_ram + description: Each image has a meaningful value for `min_ram`. + - id: scs-0102-prop-os_version + description: Each image has a meaningful value for `os_version`. + - id: scs-0102-prop-os_distro + description: Each image has a meaningful value for `os_distro`. + - id: scs-0102-prop-os_purpose + description: Each image has a meaningful value for `os_purpose`. + - id: scs-0102-prop-hw_disk_bus + description: Each image has a meaningful value for `hw_disk_bus`. + - id: scs-0102-prop-hypervisor_type + description: Each image has a meaningful value for `hypervisor_type`. + - id: scs-0102-prop-hw_rng_model + description: Each image has a meaningful value for `hw_rng_model`. + - id: scs-0102-prop-image_build_date + description: Each image has a meaningful value for `image_build_date`. + - id: scs-0102-prop-image_original_user + description: Each image has a meaningful value for `image_original_user`. + - id: scs-0102-prop-image_source + description: Each image has a meaningful value for `image_source`. + - id: scs-0102-prop-image_description + description: Each image has a meaningful value for `image_description`. + - id: scs-0102-prop-replace_frequency + description: Each image has a meaningful value for `replace_frequency`. + - id: scs-0102-prop-provided_until + description: Each image has a meaningful value for `provided_until`. + - id: scs-0102-prop-uuid_validity + description: Each image has a meaningful value for `uuid_validity`. + - id: scs-0102-prop-hotfix_hours + description: Each image has a meaningful value for `hotfix_hours`. + - id: scs-0102-image-recency + description: Each image is as recent as properties (if set) suggest. + - id: scs-0103-flavor-1v-4 + description: Check presence of flavor `SCS-1V-4` + - id: scs-0103-flavor-2v-8 + description: Check presence of flavor `SCS-2V-8` + - id: scs-0103-flavor-4v-16 + description: Check presence of flavor `SCS-4V-16` + - id: scs-0103-flavor-8v-32 + description: Check presence of flavor `SCS-8V-32` + - id: scs-0103-flavor-1v-2 + description: Check presence of flavor `SCS-1V-2` + - id: scs-0103-flavor-2v-4 + description: Check presence of flavor `SCS-2V-4` + - id: scs-0103-flavor-4v-8 + description: Check presence of flavor `SCS-4V-8` + - id: scs-0103-flavor-8v-16 + description: Check presence of flavor `SCS-8V-16` + - id: scs-0103-flavor-16v-32 + description: Check presence of flavor `SCS-16V-32` + - id: scs-0103-flavor-1v-8 + description: Check presence of flavor `SCS-1V-8` + - id: scs-0103-flavor-2v-16 + description: Check presence of flavor `SCS-2V-16` + - id: scs-0103-flavor-4v-32 + description: Check presence of flavor `SCS-4V-32` + - id: scs-0103-flavor-1l-1 + description: Check presence of flavor `SCS-1L-1` + - id: scs-0103-flavor-2v-4-20s + description: Check presence of flavor `SCS-2V-4-20s` + - id: scs-0103-flavor-4v-16-100s + description: Check presence of flavor `SCS-4V-16-100s` + - id: scs-0103-flavor-1v-4-10 + description: Check presence of flavor `SCS-1V-4-10` + - id: scs-0103-flavor-2v-8-20 + description: Check presence of flavor `SCS-2V-8-20` + - id: scs-0103-flavor-4v-16-50 + description: Check presence of flavor `SCS-4V-16-50` + - id: scs-0103-flavor-8v-32-100 + description: Check presence of flavor `SCS-8V-32-100` + - id: scs-0103-flavor-1v-2-5 + description: Check presence of flavor `SCS-1V-2-5` + - id: scs-0103-flavor-2v-4-10 + description: Check presence of flavor `SCS-2V-4-10` + - id: scs-0103-flavor-4v-8-20 + description: Check presence of flavor `SCS-4V-8-20` + - id: scs-0103-flavor-8v-16-50 + description: Check presence of flavor `SCS-8V-16-50` + - id: scs-0103-flavor-16v-32-100 + description: Check presence of flavor `SCS-16V-32-100` + - id: scs-0103-flavor-1v-8-20 + description: Check presence of flavor `SCS-1V-8-20` + - id: scs-0103-flavor-2v-16-50 + description: Check presence of flavor `SCS-2V-16-50` + - id: scs-0103-flavor-4v-32-100 + description: Check presence of flavor `SCS-4V-32-100` + - id: scs-0103-flavor-1l-1-5 + description: Check presence of flavor `SCS-1L-1-5` + - id: scs-0103-flavor-16v-64 + description: Check presence of flavor `SCS-16V-64` + - id: scs-0103-flavor-8v-64 + description: Check presence of flavor `SCS-8V-64` + - id: scs-0103-flavor-16v-128 + description: Check presence of flavor `SCS-16V-128` + - id: scs-0104-source-capi-1 + description: CAPI images adhere to canonical image source + - id: scs-0104-source-capi-2 + description: CAPI images adhere to canonical image source + - id: scs-0104-source-ubuntu-2404 + description: Ubuntu 24.04 images adhere to canonical image source + - id: scs-0104-source-ubuntu-2204 + description: Ubuntu 22.04 images adhere to canonical image source + - id: scs-0104-source-ubuntu-2004 + description: Ubuntu 20.04 images adhere to canonical image source + - id: scs-0104-source-debian-13 + description: Debian 13 images adhere to canonical image source + - id: scs-0104-source-debian-12 + description: Debian 12 images adhere to canonical image source + - id: scs-0104-source-debian-11 + description: Debian 11 images adhere to canonical image source + - id: scs-0104-image-capi-2 + description: CAPI image is present (naming scheme v2) + - id: scs-0104-image-capi-1 + description: CAPI image is present (naming scheme v1) + - id: scs-0104-image-ubuntu-2404 + description: Ubuntu 24.04 image is present (by name) + - id: scs-0104-image-ubuntu-2204 + description: Ubuntu 22.04 image is present (by name) + - id: scs-0104-image-debian-13 + description: Debian 13 image is present (by name) + - id: scs-0104-image-debian-12 + description: Debian 12 image is present (by name) + - id: scs-0114-encrypted-type + description: An encrypted volume type can be discovered. + - id: scs-0114-replicated-type + description: A replicated volume type can be discovered. + # do monolithic test for the default security groups because it's hard to decompose, and frankly, I don't + # see the correspondence between the test script and the standards requirements and recommendations + # (the latter probably simply aren't checked?) + - id: scs-0115-default-rules + description: Must fulfill all requirements of + - id: scs-0116-presence + description: Key manager service is discoverable. + - id: scs-0116-permissions + description: Key manager (if present) is usable with member role. + - id: scs-0117-test-backup + description: Check that volume backup works. + - id: scs-0123-service-compute + description: Compute service is discoverable. + - id: scs-0123-service-identity + description: Identity service is discoverable. + - id: scs-0123-service-image + description: Image service is discoverable. + - id: scs-0123-service-network + description: Network service is discoverable. + - id: scs-0123-service-load-balancer + description: Load-balancer service is discoverable. + - id: scs-0123-service-placement + description: Placement service is discoverable. + - id: scs-0123-service-object-store + description: Object-store service is discoverable. + - id: scs-0123-storage-apis + description: The block-storage API is discoverable as `volume`, `volumev3`, or `block-storage`. + - id: scs-0123-swift-s3 + description: The object-storage API is compatible with S3. +- testcases: + - id: key-manager-docs-check + description: > + Note: manual check! Must fulfill documentation requirements of . + - id: availability-zones-check + description: > + Note: manual check! Must fulfill all requirements of + - id: domain-manager-check + description: > + Note: manual check! Must fulfill all requirements of modules: - id: opc-v2022.11 name: OpenStack Powered Compute v2022.11 @@ -14,521 +203,172 @@ modules: - id: scs-0100-v3.1 name: Flavor naming v3.1 url: https://docs.scs.community/standards/scs-0100-v3-flavor-naming - run: - - executable: ./iaas/openstack_test.py - args: -c {os_cloud} scs-0100-syntax-check scs-0100-semantics-check flavor-name-check - testcases: - - id: scs-0100-syntax-check - tags: [mandatory] - description: Flavor names starting `SCS-` comply with syntax. - - id: scs-0100-semantics-check - tags: [mandatory] - description: Syntactically correct SCS flavor names represent the truth. - - id: flavor-name-check - tags: [legacy_mandatory] - description: > - Must fulfill all requirements of - + targets: + main: + - scs-0100-syntax-check + - scs-0100-semantics-check - id: scs-0101-v1 name: Entropy v1 url: https://docs.scs.community/standards/scs-0101-v1-entropy - run: - - executable: ./iaas/openstack_test.py - args: > - -c {os_cloud} - scs-0101-flavor-property scs-0101-image-property scs-0101-rngd scs-0101-entropy-avail scs-0101-fips-test - entropy-check - testcases: - - id: scs-0101-flavor-property - tags: [recommended] - description: > - Must have all flavor properties recommended in - - - id: scs-0101-image-property - tags: [recommended] - description: > - Must have all image properties recommended in - - - id: scs-0101-rngd - tags: [mandatory] - description: > - Images of the test sample must have the service `rngd`; see - - - id: scs-0101-entropy-avail - tags: [mandatory] - description: > - A test instance must have the correct `entropy_avail`; see - - - id: scs-0101-fips-test - tags: [mandatory] - description: > - A test instance must pass the "FIPS test"; see - - - id: entropy-check - tags: [legacy_mandatory] - description: > - Must fulfill all requirements of - + targets: + main: + - scs-0101-rngd + - scs-0101-entropy-avail + - scs-0101-fips-test + recommended: + - scs-0101-flavor-property + - scs-0101-image-property - id: scs-0102-v1 name: Image metadata v1 url: https://docs.scs.community/standards/scs-0102-v1-image-metadata - run: - - executable: ./iaas/openstack_test.py - args: > - -c {os_cloud} - scs-0102-prop-architecture scs-0102-prop-min_disk scs-0102-prop-min_ram - scs-0102-prop-os_version scs-0102-prop-os_distro scs-0102-prop-os_purpose scs-0102-prop-hw_disk_bus - scs-0102-prop-hypervisor_type scs-0102-prop-hw_rng_model scs-0102-prop-image_build_date - scs-0102-prop-image_original_user scs-0102-prop-image_source scs-0102-prop-image_description - scs-0102-prop-replace_frequency scs-0102-prop-provided_until scs-0102-prop-uuid_validity - scs-0102-prop-hotfix_hours scs-0102-image-recency scs-0102-prop-hash_algo - image-metadata-check - testcases: - - id: scs-0102-prop-architecture - tags: [mandatory] - description: Each image has a meaningful value for `architecture`. - - id: scs-0102-prop-hash_algo - tags: [recommended] - description: Each image has a meaningful value for `hash_algo`. - - id: scs-0102-prop-min_disk - tags: [mandatory] - description: Each image has a meaningful value for `min_disk`. - - id: scs-0102-prop-min_ram - tags: [mandatory] - description: Each image has a meaningful value for `min_ram`. - - id: scs-0102-prop-os_version - tags: [mandatory] - description: Each image has a meaningful value for `os_version`. - - id: scs-0102-prop-os_distro - tags: [mandatory] - description: Each image has a meaningful value for `os_distro`. - - id: scs-0102-prop-os_purpose - tags: [recommended] - description: Each image has a meaningful value for `os_purpose`. - - id: scs-0102-prop-hw_disk_bus - tags: [mandatory] - description: Each image has a meaningful value for `hw_disk_bus`. - - id: scs-0102-prop-hypervisor_type - tags: [recommended] - description: Each image has a meaningful value for `hypervisor_type`. - - id: scs-0102-prop-hw_rng_model - tags: [recommended] - description: Each image has a meaningful value for `hw_rng_model`. - - id: scs-0102-prop-image_build_date - tags: [mandatory] - description: Each image has a meaningful value for `image_build_date`. - - id: scs-0102-prop-image_original_user - tags: [mandatory] - description: Each image has a meaningful value for `image_original_user`. - - id: scs-0102-prop-image_source - tags: [mandatory] - description: Each image has a meaningful value for `image_source`. - - id: scs-0102-prop-image_description - tags: [mandatory] - description: Each image has a meaningful value for `image_description`. - - id: scs-0102-prop-replace_frequency - tags: [mandatory] - description: Each image has a meaningful value for `replace_frequency`. - - id: scs-0102-prop-provided_until - tags: [mandatory] - description: Each image has a meaningful value for `provided_until`. - - id: scs-0102-prop-uuid_validity - tags: [mandatory] - description: Each image has a meaningful value for `uuid_validity`. - - id: scs-0102-prop-hotfix_hours - tags: [recommended] - description: Each image has a meaningful value for `hotfix_hours`. - - id: scs-0102-image-recency - tags: [mandatory] - description: Each image is as recent as properties (if set) suggest. - - id: image-metadata-check - tags: [legacy_mandatory] - description: > - Must fulfill all requirements of + targets: + main: + - scs-0102-prop-architecture + - scs-0102-prop-min_disk + - scs-0102-prop-min_ram + - scs-0102-prop-os_version + - scs-0102-prop-os_distro + - scs-0102-prop-hw_disk_bus + - scs-0102-prop-image_build_date + - scs-0102-prop-image_original_user + - scs-0102-prop-image_source + - scs-0102-prop-image_description + - scs-0102-prop-replace_frequency + - scs-0102-prop-provided_until + - scs-0102-prop-uuid_validity + - scs-0102-image-recency + recommended: + - scs-0102-prop-hash_algo + - scs-0102-prop-os_purpose + - scs-0102-prop-hypervisor_type + - scs-0102-prop-hw_rng_model + - scs-0102-prop-hotfix_hours - id: scs-0103-v1 name: Standard flavors url: https://docs.scs.community/standards/scs-0103-v1-standard-flavors - run: - - executable: ./iaas/openstack_test.py - args: > - -c {os_cloud} - scs-0103-flavor-1v-4 scs-0103-flavor-2v-8 scs-0103-flavor-4v-16 scs-0103-flavor-8v-32 - scs-0103-flavor-1v-2 scs-0103-flavor-2v-4 scs-0103-flavor-4v-8 scs-0103-flavor-8v-16 scs-0103-flavor-16v-32 - scs-0103-flavor-1v-8 scs-0103-flavor-2v-16 scs-0103-flavor-4v-32 scs-0103-flavor-1l-1 - scs-0103-flavor-2v-4-20s scs-0103-flavor-4v-16-100s - scs-0103-flavor-1v-4-10 scs-0103-flavor-2v-8-20 scs-0103-flavor-4v-16-50 scs-0103-flavor-8v-32-100 - scs-0103-flavor-1v-2-5 scs-0103-flavor-2v-4-10 scs-0103-flavor-4v-8-20 scs-0103-flavor-8v-16-50 - scs-0103-flavor-16v-32-100 scs-0103-flavor-1v-8-20 scs-0103-flavor-2v-16-50 scs-0103-flavor-4v-32-100 - scs-0103-flavor-1l-1-5 scs-0103-flavor-16v-64 scs-0103-flavor-8v-64 scs-0103-flavor-16v-128 - standard-flavors-check - testcases: - - id: scs-0103-flavor-1v-4 - tags: [mandatory] - description: Check presence of flavor `SCS-1V-4` - - id: scs-0103-flavor-2v-8 - tags: [mandatory] - description: Check presence of flavor `SCS-2V-8` - - id: scs-0103-flavor-4v-16 - tags: [mandatory] - description: Check presence of flavor `SCS-4V-16` - - id: scs-0103-flavor-8v-32 - tags: [mandatory] - description: Check presence of flavor `SCS-8V-32` - - id: scs-0103-flavor-1v-2 - tags: [mandatory] - description: Check presence of flavor `SCS-1V-2` - - id: scs-0103-flavor-2v-4 - tags: [mandatory] - description: Check presence of flavor `SCS-2V-4` - - id: scs-0103-flavor-4v-8 - tags: [mandatory] - description: Check presence of flavor `SCS-4V-8` - - id: scs-0103-flavor-8v-16 - tags: [mandatory] - description: Check presence of flavor `SCS-8V-16` - - id: scs-0103-flavor-16v-32 - tags: [mandatory] - description: Check presence of flavor `SCS-16V-32` - - id: scs-0103-flavor-1v-8 - tags: [mandatory] - description: Check presence of flavor `SCS-1V-8` - - id: scs-0103-flavor-2v-16 - tags: [mandatory] - description: Check presence of flavor `SCS-2V-16` - - id: scs-0103-flavor-4v-32 - tags: [mandatory] - description: Check presence of flavor `SCS-4V-32` - - id: scs-0103-flavor-1l-1 - tags: [mandatory] - description: Check presence of flavor `SCS-1L-1` - - id: scs-0103-flavor-2v-4-20s - tags: [mandatory] - description: Check presence of flavor `SCS-2V-4-20s` - - id: scs-0103-flavor-4v-16-100s - tags: [mandatory] - description: Check presence of flavor `SCS-4V-16-100s` - - id: scs-0103-flavor-1v-4-10 - tags: [recommended] - description: Check presence of flavor `SCS-1V-4-10` - - id: scs-0103-flavor-2v-8-20 - tags: [recommended] - description: Check presence of flavor `SCS-2V-8-20` - - id: scs-0103-flavor-4v-16-50 - tags: [recommended] - description: Check presence of flavor `SCS-4V-16-50` - - id: scs-0103-flavor-8v-32-100 - tags: [recommended] - description: Check presence of flavor `SCS-8V-32-100` - - id: scs-0103-flavor-1v-2-5 - tags: [recommended] - description: Check presence of flavor `SCS-1V-2-5` - - id: scs-0103-flavor-2v-4-10 - tags: [recommended] - description: Check presence of flavor `SCS-2V-4-10` - - id: scs-0103-flavor-4v-8-20 - tags: [recommended] - description: Check presence of flavor `SCS-4V-8-20` - - id: scs-0103-flavor-8v-16-50 - tags: [recommended] - description: Check presence of flavor `SCS-8V-16-50` - - id: scs-0103-flavor-16v-32-100 - tags: [recommended] - description: Check presence of flavor `SCS-16V-32-100` - - id: scs-0103-flavor-1v-8-20 - tags: [recommended] - description: Check presence of flavor `SCS-1V-8-20` - - id: scs-0103-flavor-2v-16-50 - tags: [recommended] - description: Check presence of flavor `SCS-2V-16-50` - - id: scs-0103-flavor-4v-32-100 - tags: [recommended] - description: Check presence of flavor `SCS-4V-32-100` - - id: scs-0103-flavor-1l-1-5 - tags: [recommended] - description: Check presence of flavor `SCS-1L-1-5` - - id: scs-0103-flavor-16v-64 - tags: [recommended] - description: Check presence of flavor `SCS-16V-64` - - id: scs-0103-flavor-8v-64 - tags: [recommended] - description: Check presence of flavor `SCS-8V-64` - - id: scs-0103-flavor-16v-128 - tags: [recommended] - description: Check presence of flavor `SCS-16V-128` - - id: standard-flavors-check - tags: [legacy_mandatory] - description: > - Must fulfill all requirements of + targets: + main: + - scs-0103-flavor-1v-4 + - scs-0103-flavor-2v-8 + - scs-0103-flavor-4v-16 + - scs-0103-flavor-8v-32 + - scs-0103-flavor-1v-2 + - scs-0103-flavor-2v-4 + - scs-0103-flavor-4v-8 + - scs-0103-flavor-8v-16 + - scs-0103-flavor-16v-32 + - scs-0103-flavor-1v-8 + - scs-0103-flavor-2v-16 + - scs-0103-flavor-4v-32 + - scs-0103-flavor-1l-1 + - scs-0103-flavor-2v-4-20s + - scs-0103-flavor-4v-16-100s + recommended: + - scs-0103-flavor-1v-4-10 + - scs-0103-flavor-2v-8-20 + - scs-0103-flavor-4v-16-50 + - scs-0103-flavor-8v-32-100 + - scs-0103-flavor-1v-2-5 + - scs-0103-flavor-2v-4-10 + - scs-0103-flavor-4v-8-20 + - scs-0103-flavor-8v-16-50 + - scs-0103-flavor-16v-32-100 + - scs-0103-flavor-1v-8-20 + - scs-0103-flavor-2v-16-50 + - scs-0103-flavor-4v-32-100 + - scs-0103-flavor-1l-1-5 + - scs-0103-flavor-16v-64 + - scs-0103-flavor-8v-64 + - scs-0103-flavor-16v-128 - id: scs-0104-v1-1 name: Standard images url: https://docs.scs.community/standards/scs-0104-v1-standard-images parameters: image_spec: address (URL) of an image-spec (YAML) file - run: - - executable: ./iaas/openstack_test.py - args: > - -c {os_cloud} - scs-0104-source-capi-1 scs-0104-source-capi-2 - scs-0104-source-ubuntu-2404 scs-0104-source-ubuntu-2204 scs-0104-source-ubuntu-2004 - scs-0104-source-debian-13 scs-0104-source-debian-12 scs-0104-source-debian-11 - scs-0104-image-capi-2 scs-0104-image-capi-1 - scs-0104-image-ubuntu-2404 scs-0104-image-ubuntu-2204 - scs-0104-image-debian-13 scs-0104-image-debian-12 - standard-images-check/1 - testcases: - - id: scs-0104-source-capi-1 - tags: [mandatory] - description: CAPI images adhere to canonical image source - - id: scs-0104-source-capi-2 - tags: [mandatory] - description: CAPI images adhere to canonical image source - - id: scs-0104-source-ubuntu-2404 - tags: [mandatory] - description: Ubuntu 24.04 images adhere to canonical image source - - id: scs-0104-source-ubuntu-2204 - tags: [mandatory] - description: Ubuntu 22.04 images adhere to canonical image source - - id: scs-0104-source-ubuntu-2004 - tags: [mandatory] - description: Ubuntu 20.04 images adhere to canonical image source - - id: scs-0104-source-debian-13 - tags: [mandatory] - description: Debian 13 images adhere to canonical image source - - id: scs-0104-source-debian-12 - tags: [mandatory] - description: Debian 12 images adhere to canonical image source - - id: scs-0104-source-debian-11 - tags: [mandatory] - description: Debian 11 images adhere to canonical image source - - id: scs-0104-image-capi-2 - tags: [] - description: CAPI image is present (naming scheme v2) - - id: scs-0104-image-capi-1 - tags: [recommended] - description: CAPI image is present (naming scheme v1) - - id: scs-0104-image-ubuntu-2404 - tags: [] - description: Ubuntu 24.04 image is present (by name) - - id: scs-0104-image-ubuntu-2204 - tags: [mandatory] - description: Ubuntu 22.04 image is present (by name) - - id: scs-0104-image-debian-13 - tags: [] - description: Debian 13 image is present (by name) - - id: scs-0104-image-debian-12 - tags: [] - description: Debian 12 image is present (by name) - - id: standard-images-check - tags: [legacy_mandatory] - description: > - Must fulfill all requirements of + targets: + main: + - scs-0104-source-capi-1 + - scs-0104-source-ubuntu-2204 + - scs-0104-source-ubuntu-2004 + - scs-0104-source-debian-12 + - scs-0104-source-debian-11 + - scs-0104-image-ubuntu-2204 + recommended: + - scs-0104-image-capi-1 - id: scs-0104-v1-2 name: Standard images url: https://docs.scs.community/standards/scs-0104-v1-standard-images parameters: image_spec: address (URL) of an image-spec (YAML) file - run: - - executable: ./iaas/openstack_test.py - args: > - -c {os_cloud} - scs-0104-source-capi-1 scs-0104-source-capi-2 - scs-0104-source-ubuntu-2404 scs-0104-source-ubuntu-2204 scs-0104-source-ubuntu-2004 - scs-0104-source-debian-13 scs-0104-source-debian-12 scs-0104-source-debian-11 - scs-0104-image-capi-2 scs-0104-image-capi-1 - scs-0104-image-ubuntu-2404 scs-0104-image-ubuntu-2204 - scs-0104-image-debian-13 scs-0104-image-debian-12 - standard-images-check/2 - testcases: - - id: scs-0104-source-capi-1 - tags: [mandatory] - description: CAPI images adhere to canonical image source - - id: scs-0104-source-capi-2 - tags: [mandatory] - description: CAPI images adhere to canonical image source - - id: scs-0104-source-ubuntu-2404 - tags: [mandatory] - description: Ubuntu 24.04 images adhere to canonical image source - - id: scs-0104-source-ubuntu-2204 - tags: [mandatory] - description: Ubuntu 22.04 images adhere to canonical image source - - id: scs-0104-source-ubuntu-2004 - tags: [mandatory] - description: Ubuntu 20.04 images adhere to canonical image source - - id: scs-0104-source-debian-13 - tags: [mandatory] - description: Debian 13 images adhere to canonical image source - - id: scs-0104-source-debian-12 - tags: [mandatory] - description: Debian 12 images adhere to canonical image source - - id: scs-0104-source-debian-11 - tags: [mandatory] - description: Debian 11 images adhere to canonical image source - - id: scs-0104-image-capi-2 - tags: [recommended] - description: CAPI image is present (naming scheme v2) - - id: scs-0104-image-capi-1 - tags: [] - description: CAPI image is present (naming scheme v1) - - id: scs-0104-image-ubuntu-2404 - tags: [mandatory] - description: Ubuntu 24.04 image is present (by name) - - id: scs-0104-image-ubuntu-2204 - tags: [] - description: Ubuntu 22.04 image is present (by name) - - id: scs-0104-image-debian-13 - tags: [] - description: Debian 13 image is present (by name) - - id: scs-0104-image-debian-12 - tags: [recommended] - description: Debian 12 image is present (by name) - - id: standard-images-check - tags: [legacy_mandatory] - description: > - Must fulfill all requirements of + targets: + main: + - scs-0104-source-capi-1 + - scs-0104-source-capi-2 + - scs-0104-source-ubuntu-2404 + - scs-0104-source-ubuntu-2204 + - scs-0104-source-ubuntu-2004 + - scs-0104-source-debian-13 + - scs-0104-source-debian-12 + - scs-0104-source-debian-11 + - scs-0104-image-ubuntu-2404 + recommended: + - scs-0104-image-capi-2 + - scs-0104-image-debian-12 - id: scs-0114-v1 name: Volume Types url: https://docs.scs.community/standards/scs-0114-v1-volume-type-standard - run: - - executable: ./iaas/openstack_test.py - args: > - -c {os_cloud} - scs-0114-encrypted-type scs-0114-replicated-type - volume-types-check - testcases: - - id: scs-0114-encrypted-type - tags: [recommended] - description: An encrypted volume type can be discovered. - - id: scs-0114-replicated-type - tags: [recommended] - description: A replicated volume type can be discovered. - - id: volume-types-check - tags: [recommended] - description: > - Must fulfill all requirements of + targets: + recommended: + - scs-0114-encrypted-type + - scs-0114-replicated-type - id: scs-0115-v1 name: Default rules for security groups url: https://docs.scs.community/standards/scs-0115-v1-default-rules-for-security-groups - run: - - executable: ./iaas/openstack_test.py - args: > - -c {os_cloud} - scs-0115-default-rules - security-groups-default-rules-check - testcases: - # do monolithic test for the default security groups because it's hard to decompose, and frankly, I don't - # see the correspondence between the test script and the standards requirements and recommendations - # (the latter probably simply aren't checked?) - - id: scs-0115-default-rules - tags: [mandatory] - description: > - Must fulfill all requirements of - - id: security-groups-default-rules-check - tags: [legacy_mandatory] - description: > - Must fulfill all requirements of + targets: + main: + - scs-0115-default-rules - id: scs-0116-v1 name: Key manager url: https://docs.scs.community/standards/scs-0116-v1-key-manager-standard - run: - - executable: ./iaas/openstack_test.py - args: > - -c {os_cloud} - scs-0116-presence scs-0116-permissions - key-manager-check - testcases: - - id: scs-0116-presence - tags: [recommended] - description: Key manager service is discoverable. - - id: scs-0116-permissions - tags: [mandatory] - description: Key manager (if present) is usable with member role. - - id: key-manager-check - tags: [legacy_mandatory] - description: > - Must fulfill all requirements of - - id: key-manager-docs-check - tags: [key-manager-docs] - description: > - Note: manual check! Must fulfill documentation requirements of . + targets: + main: + - scs-0116-permissions + recommended: + - scs-0116-presence + preview: + - key-manager-docs-check - id: scs-0117-v1 name: Volume backup url: https://docs.scs.community/standards/scs-0117-v1-volume-backup-service - run: - - executable: ./iaas/openstack_test.py - args: > - -c {os_cloud} - scs-0117-test-backup - volume-backup-check - testcases: - - id: scs-0117-test-backup - tags: [mandatory] - description: Check that volume backup works. - - id: volume-backup-check - tags: [legacy_mandatory] - description: > - Must fulfill all requirements of + targets: + main: + - scs-0117-test-backup - id: scs-0121-v1 name: Availability Zones url: https://docs.scs.community/standards/scs-0121-v1-Availability-Zones-Standard - testcases: - - id: availability-zones-check - tags: [availability-zones] - description: > - Note: manual check! Must fulfill all requirements of + targets: + preview: + - availability-zones-check - id: scs-0123-v1 name: Mandatory and Supported IaaS Services url: https://docs.scs.community/standards/scs-0123-v1-mandatory-and-supported-IaaS-services - run: - - executable: ./iaas/openstack_test.py - args: > - -c {os_cloud} - scs-0123-service-compute scs-0123-service-identity scs-0123-service-image scs-0123-service-network - scs-0123-service-load-balancer scs-0123-service-placement scs-0123-service-object-store - scs-0123-storage-apis scs-0123-swift-s3 - service-apis-check - testcases: - - id: scs-0123-service-compute - tags: [mandatory] - description: Compute service is discoverable. - - id: scs-0123-service-identity - tags: [mandatory] - description: Identity service is discoverable. - - id: scs-0123-service-image - tags: [mandatory] - description: Image service is discoverable. - - id: scs-0123-service-network - tags: [mandatory] - description: Network service is discoverable. - - id: scs-0123-service-load-balancer - tags: [mandatory] - description: Load-balancer service is discoverable. - - id: scs-0123-service-placement - tags: [mandatory] - description: Placement service is discoverable. - - id: scs-0123-service-object-store - tags: [mandatory] - description: Object-store service is discoverable. - - id: scs-0123-storage-apis - tags: [mandatory] - description: The block-storage API is discoverable as `volume`, `volumev3`, or `block-storage`. - - id: scs-0123-swift-s3 - tags: [mandatory] - description: The object-storage API is compatible with S3. - - id: service-apis-check - tags: [legacy_mandatory] - description: > - Must fulfill all requirements of (except for documentation requirements, which are tested manually with service-apis-docs-check). + targets: + main: + - scs-0123-service-compute + - scs-0123-service-identity + - scs-0123-service-image + - scs-0123-service-network + - scs-0123-service-load-balancer + - scs-0123-service-placement + - scs-0123-service-object-store + - scs-0123-storage-apis + - scs-0123-swift-s3 - id: scs-0302-v1 name: Domain Manager Role url: https://docs.scs.community/standards/scs-0302-v1-domain-manager-role - # run: - # - executable: ./iam/domain-manager/domain-manager-check.py - # args: --os-cloud {os_cloud} --debug --domain-config ... - testcases: - - id: domain-manager-check - tags: [domain-manager] - description: > - Note: manual check! Must fulfill all requirements of + targets: + preview: + - domain-manager-check timeline: - date: 2025-09-09 versions: @@ -569,10 +409,6 @@ versions: - scs-0121-v1 - scs-0123-v1 - scs-0302-v1 - targets: - main: mandatory - recommended: recommended - preview: domain-manager/availability-zones/key-manager-docs - version: v5.1 # copy of v5, but with include "scs-0123-v1", which had simply been forgotten stabilized_at: 2024-12-19 include: @@ -591,9 +427,6 @@ versions: - scs-0121-v1 - scs-0123-v1 - scs-0302-v1 - targets: - main: mandatory - preview: domain-manager/availability-zones/key-manager-docs - version: v4 stabilized_at: 2024-02-28 include: @@ -605,8 +438,6 @@ versions: - ref: scs-0104-v1-1 parameters: image_spec: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/iaas/scs-0104-v1-images.yaml - targets: - main: mandatory - version: v3 # comment: > # This is what our documentation wrongly stated as being v3 when we introduced v4. @@ -616,5 +447,3 @@ versions: - opc-v2022.11 - scs-0100-v3.1 - scs-0102-v1 - targets: - main: mandatory diff --git a/Tests/scs-compatible-kaas.yaml b/Tests/scs-compatible-kaas.yaml index a1e595aed..4929bc906 100644 --- a/Tests/scs-compatible-kaas.yaml +++ b/Tests/scs-compatible-kaas.yaml @@ -9,53 +9,58 @@ variables: # working directory for the subject under test # (note that we consider each kubernetes branch a test subject of its own) - kubeconfig +scripts: +- executable: ./kaas/sonobuoy_handler/run_sonobuoy.py + args: -k {kubeconfig} -r {subject_root}/sono-results -c 'cncf-k8s-conformance' -a '--mode=certified-conformance' + #~ args: -k {kubeconfig} -r {subject_root}/sono-results -c 'cncf-k8s-conformance' -a '--plugin-env e2e.E2E_DRYRUN=true' + testcases: + - id: cncf-k8s-conformance + lifetime: year + description: > + Must fulfill all requirements of [CNCF Kubernetes conformance](https://github.com/cncf/k8s-conformance/tree/master) +- executable: ./kaas/k8s-version-policy/k8s_version_policy.py + args: -k {kubeconfig} + testcases: + - id: version-policy-check + description: > + Must fulfill all requirements of +- executable: ./kaas/k8s-node-distribution/k8s_node_distribution_check.py + args: -k {kubeconfig} + testcases: + - id: node-distribution-check + description: > + Must fulfill all requirements of +- executable: ./kaas/sonobuoy_handler/run_sonobuoy.py + args: -k {kubeconfig} -r {subject_root}/sono-results -c 'kaas-networking-check' -a '--e2e-focus "NetworkPolicy"' + testcases: + - id: kaas-networking-check + description: > + Must fulfill all requirements of modules: - id: cncf-k8s-conformance name: CNCF Kubernetes conformance url: https://github.com/cncf/k8s-conformance/tree/master - run: - - executable: ./kaas/sonobuoy_handler/run_sonobuoy.py - args: -k {kubeconfig} -r {subject_root}/sono-results -c 'cncf-k8s-conformance' -a '--mode=certified-conformance' - #~ args: -k {kubeconfig} -r {subject_root}/sono-results -c 'cncf-k8s-conformance' -a '--plugin-env e2e.E2E_DRYRUN=true' - testcases: - - id: cncf-k8s-conformance - tags: [mandatory] - lifetime: year - description: > - Must fulfill all requirements of [CNCF Kubernetes conformance](https://github.com/cncf/k8s-conformance/tree/master) + targets: + main: + - cncf-k8s-conformance - id: scs-0210-v2 name: Kubernetes version policy url: https://docs.scs.community/standards/scs-0210-v2-k8s-version-policy - run: - - executable: ./kaas/k8s-version-policy/k8s_version_policy.py - args: -k {kubeconfig} - testcases: - - id: version-policy-check - tags: [mandatory] - description: > - Must fulfill all requirements of + targets: + main: + - version-policy-check - id: scs-0214-v2 name: Kubernetes node distribution and availability url: https://docs.scs.community/standards/scs-0214-v2-k8s-node-distribution - run: - - executable: ./kaas/k8s-node-distribution/k8s_node_distribution_check.py - args: -k {kubeconfig} - testcases: - - id: node-distribution-check - tags: [mandatory] - description: > - Must fulfill all requirements of + targets: + main: + - node-distribution-check - id: scs-0219-v1 name: KaaS networking url: https://docs.scs.community/standards/scs-0219-v1-kaas-networking - run: - - executable: ./kaas/sonobuoy_handler/run_sonobuoy.py - args: -k {kubeconfig} -r {subject_root}/sono-results -c 'kaas-networking-check' -a '--e2e-focus "NetworkPolicy"' - testcases: - - id: kaas-networking-check - tags: [mandatory] - description: > - Must fulfill all requirements of + targets: + main: + - kaas-networking-check timeline: - date: 2024-11-26 versions: @@ -71,5 +76,3 @@ versions: - scs-0210-v2 - scs-0214-v2 - scs-0219-v1 - targets: - main: mandatory diff --git a/Tests/scs-compliance-check.py b/Tests/scs-compliance-check.py index 315794259..49458d573 100755 --- a/Tests/scs-compliance-check.py +++ b/Tests/scs-compliance-check.py @@ -5,7 +5,7 @@ # # (c) Eduard Itrich # (c) Kurt Garloff -# (c) Matthias Büchse +# (c) Matthias Büchse # SPDX-License-Identifier: Apache-2.0 """Main SCS compliance checker @@ -26,12 +26,11 @@ import getopt import datetime import subprocess -from collections import defaultdict from itertools import chain import logging import yaml -from scs_cert_lib import load_spec, annotate_validity, compile_suite, TestSuite, TESTCASE_VERDICTS +from scs_cert_lib import load_spec, annotate_validity, eval_buckets, TESTCASE_VERDICTS logger = logging.getLogger(__name__) @@ -153,10 +152,6 @@ def select_valid(versions: list) -> list: return [version for version in versions if version['_explicit_validity']] -def suppress(*args, **kwargs): - return - - def invoke_check_tool(exe, args, env, cwd): """run check tool and return invokation dict to use in the report""" try: @@ -182,7 +177,7 @@ def invoke_check_tool(exe, args, env, cwd): return invokation -def compute_results(stdout): +def compute_results(stdout, permissible_ids=()): """pick out test results from stdout lines""" result = {} for line in stdout: @@ -192,7 +187,11 @@ def compute_results(stdout): value = TESTCASE_VERDICTS.get(parts[1].strip().upper()) if value is None: continue - result[parts[0].strip()] = value + testcase_id = parts[0].strip() + if permissible_ids and testcase_id not in permissible_ids: + logger.warning(f"ignoring invalid result id: {testcase_id}") + continue + result[testcase_id] = value return result @@ -200,39 +199,34 @@ class CheckRunner: def __init__(self, cwd, assignment, verbosity=0): self.cwd = cwd self.assignment = assignment - self.memo = {} self.num_abort = 0 self.num_error = 0 self.verbosity = verbosity self.spamminess = 0 - def run(self, check): - parameters = check.get('parameters') - assignment = {**self.assignment, **parameters} if parameters else self.assignment + def run(self, check, testcases=()): + parameters = check.get('parameters', {}) + assignment = {'testcases': ' '.join(testcases), **self.assignment, **parameters} args = check.get('args', '').format(**assignment) env = {key: value.format(**assignment) for key, value in check.get('env', {}).items()} env_str = " ".join(f"{key}={value}" for key, value in env.items()) - memo_key = f"{env_str} {check['executable']} {args}".strip() - logger.debug(f"running {memo_key!r}...") - invocation = self.memo.get(memo_key) - if invocation is None: - check_env = {**os.environ, **env} - invocation = invoke_check_tool(check["executable"], args, check_env, self.cwd) - invocation = { - 'id': str(uuid.uuid4()), - 'cmd': memo_key, - 'result': 0, # keep this for backwards compatibility - 'results': compute_results(invocation['stdout']), - **invocation - } - if self.verbosity > 1 and invocation["stdout"]: - print("\n".join(invocation["stdout"])) - self.spamminess += 1 - # the following check used to be "> 0", but this is quite verbose... - if invocation['rc'] or self.verbosity > 1 and invocation["stderr"]: - print("\n".join(invocation["stderr"])) - self.spamminess += 1 - self.memo[memo_key] = invocation + cmd = f"{env_str} {check['executable']} {args}".strip() + logger.debug(f"running {cmd!r}...") + check_env = {**os.environ, **env} + invocation = invoke_check_tool(check["executable"], args, check_env, self.cwd) + invocation = { + 'id': str(uuid.uuid4()), + 'cmd': cmd, + 'results': compute_results(invocation['stdout'], permissible_ids=testcases), + **invocation + } + if self.verbosity > 1 and invocation["stdout"]: + print("\n".join(invocation["stdout"])) + self.spamminess += 1 + # the following check used to be "> 0", but this is quite verbose... + if invocation['rc'] or self.verbosity > 1 and invocation["stderr"]: + print("\n".join(invocation["stderr"])) + self.spamminess += 1 logger.debug(f".. rc {invocation['rc']}, {invocation['critical']} critical, {invocation['error']} error") self.num_abort += invocation["critical"] self.num_error += invocation["error"] @@ -240,49 +234,10 @@ def run(self, check): self.num_error + len([value for value in invocation['results'].values() if value < 0]) return invocation - def get_invocations(self): - return {invocation['id']: invocation for invocation in self.memo.values()} - -class ResultBuilder: - def __init__(self, name): - self.name = name - self._raw = defaultdict(list) - - def record(self, id_, **kwargs): - self._raw[id_].append(kwargs) - - def finalize(self, permissible_ids=None): - final = {} - for id_, ls in self._raw.items(): - if permissible_ids is not None and id_ not in permissible_ids: - logger.warning(f"ignoring invalid result id: {id_}") - continue - # just in case: sort by value (worst first) - ls.sort(key=lambda item: item['result']) - winner, runnerups = ls[0], ls[1:] - if runnerups: - logger.warning(f"multiple result values for {id_}") - winner = {**winner, 'runnerups': runnerups} - final[id_] = winner - return final - - -def run_suite(suite: TestSuite, runner: CheckRunner): - """run all checks of `suite` using `runner`, returning results dict via `ResultBuilder`""" - suite.check_sanity() - builder = ResultBuilder(suite.name) - for check in suite.checks: - invocation = runner.run(check) - for id_, value in invocation["results"].items(): - builder.record(id_, result=value, invocation=invocation['id']) - return builder.finalize(permissible_ids=suite.ids) - - -def print_report(subject: str, suite: TestSuite, targets: dict, results: dict, verbose=False): - print(f"{subject} {suite.name}:") - for tname, target_spec in targets.items(): - by_value = suite.select(tname, target_spec).eval_buckets(results) +def print_report(testcase_lookup: dict, targets: dict, results: dict, partial=False, verbose=False): + for tname, tc_ids in targets.items(): + by_value = eval_buckets(results, tc_ids) missing, failed, aborted, passed = by_value[None], by_value[-1], by_value[0], by_value[1] verdict = 'FAIL' if failed or aborted else 'TENTATIVE pass' if missing else 'PASS' summary_parts = [f"{len(passed)} passed"] @@ -298,20 +253,23 @@ def print_report(subject: str, suite: TestSuite, targets: dict, results: dict, v if verbose: reportcateg.append((passed, 'PASSED')) for offenders, category in reportcateg: - if category == 'MISSING' and suite.partial: + if category == 'MISSING' and partial: continue # do not report each missing testcase if a filter was used if not offenders: continue print(f" - {category}:") - for testcase in offenders: - print(f" - {testcase['id']}:") + for tc_id in offenders: + print(f" - {tc_id}:") + testcase = testcase_lookup[tc_id] if 'description' in testcase: # used to be `verbose and ...`, but users need the URL! print(f" > {testcase['description'].strip()}") -def create_report(argv, config, spec, versions, invocations): +def create_report(argv, config, spec, invocations): return { # these fields are essential: + # results are no longer specific to version! + # omit the field `version` because it's just redundant; simply parse invocations (see below) "spec": { "uuid": spec['uuid'], "name": spec['name'], @@ -320,7 +278,6 @@ def create_report(argv, config, spec, versions, invocations): "checked_at": datetime.datetime.now(), "reference_date": config.checkdate, "subject": config.subject, - "versions": versions, # this field is mostly for debugging: "run": { "uuid": str(uuid.uuid4()), @@ -329,7 +286,7 @@ def create_report(argv, config, spec, versions, invocations): "sections": config.sections, "forced_version": config.version or None, "forced_tests": None if config.tests is None else config.tests.pattern, - "invocations": invocations, + "invocations": {invocation['id']: invocation for invocation in invocations}, }, } @@ -359,28 +316,54 @@ def main(argv): raise RuntimeError(f"No valid version found for {config.checkdate}") check_cwd = os.path.dirname(config.arg0) or os.getcwd() runner = CheckRunner(check_cwd, config.assignment, verbosity=config.verbose and 2 or not config.quiet) - version_report = {} - # collect report data as tuples (version, suite, results) before printing them - report_data = [] + title, partial = spec['name'], False + if config.sections: + title += f" [sections: {', '.join(config.sections)}]" + partial = True + if config.tests: + title += f" [tests: '{config.tests.pattern}']" + partial = True + # collect all testcases we need + all_testcase_ids = set() for version in versions: - vname = version['version'] - suite = compile_suite( - f"{spec['name']} {vname} ({version['validity']})", - version['include'], - config.sections, - config.tests, - ) - report_data.append((version, suite, run_suite(suite, runner))) + for testcase_ids in version['targets'].values(): + all_testcase_ids.update(testcase_ids) + # collect scripts to be run + testcase_lookup = spec['testcases'] + tc_script_lookup = spec['tc_scripts'] + script_info = {} + for tc_id in all_testcase_ids: + script = tc_script_lookup[tc_id] + if 'executable' not in script: + continue # manual check + if config.sections and script.get('section') not in config.sections: + continue + if config.tests and not config.tests.match(tc_id): + continue + item = script_info.get(id(script)) + if item is None: + _, testcase_ids = script_info[id(script)] = (script, []) + else: + _, testcase_ids = item + testcase_ids.append(tc_id) + # run scripts + invocations = [ + runner.run(script, testcases=sorted(testcases)) + for script, testcases in script_info.values() + ] + results = {} + for invocation in invocations: + results.update(invocation['results']) # now report: to console if requested, and likewise for yaml output if not config.quiet: # print a horizontal line if we had any script output if runner.spamminess: print("********" * 10) # 80 characters - for version, suite, results in report_data: - print_report(config.subject, suite, version['targets'], results, config.verbose) + for version in versions: + print(f"{config.subject} {title} {version['version']}:") + print_report(testcase_lookup, version['targets'], results, partial, config.verbose) if config.output: - version_report = {version['version']: results for version, _, results in report_data} - report = create_report(argv, config, spec, version_report, runner.get_invocations()) + report = create_report(argv, config, spec, invocations) with open(config.output, 'w', encoding='UTF-8') as fileobj: yaml.safe_dump(report, fileobj, default_flow_style=False, sort_keys=False, explicit_start=True) return min(127, runner.num_abort + (0 if config.critical_only else runner.num_error)) diff --git a/Tests/scs_cert_lib.py b/Tests/scs_cert_lib.py index 25957bc20..a07c4a30a 100644 --- a/Tests/scs_cert_lib.py +++ b/Tests/scs_cert_lib.py @@ -6,20 +6,19 @@ # (c) Matthias Büchse # SPDX-License-Identifier: Apache-2.0 -from collections import Counter, defaultdict +from collections import defaultdict from datetime import datetime, date, timedelta import logging -import re logger = logging.getLogger(__name__) # valid keywords for various parts of the spec, to be checked using `check_keywords` KEYWORDS = { - 'spec': ('uuid', 'name', 'url', 'versions', 'prerequisite', 'variables', 'modules', 'timeline'), + 'spec': ('uuid', 'name', 'url', 'versions', 'prerequisite', 'variables', 'scripts', 'modules', 'timeline'), + 'scripts': ('executable', 'env', 'args', 'section', 'testcases'), 'versions': ('version', 'include', 'targets', 'stabilized_at'), - 'modules': ('id', 'run', 'testcases', 'url', 'name', 'parameters'), - 'run': ('executable', 'env', 'args', 'section'), + 'modules': ('id', 'targets', 'url', 'name', 'parameters'), 'testcases': ('lifetime', 'id', 'description', 'tags'), 'include': ('ref', 'parameters'), } @@ -58,6 +57,15 @@ def _resolve_spec(spec: dict): # - modules, referenced by id # - versions, referenced by name (unfortunately, the field is called "version") # step 1. build lookups + testcase_lookup = {} + tc_script_lookup = {} + for script in spec.get('scripts', ()): + for testcase in script.get('testcases', ()): + id_ = testcase['id'] + if id_ in testcase_lookup: + raise RuntimeError(f"duplicate testcase {id_}") + testcase_lookup[id_] = testcase + tc_script_lookup[id_] = script module_lookup = {module['id']: module for module in spec['modules']} version_lookup = {version['version']: version for version in spec['versions']} # step 2. check for duplicates: @@ -68,6 +76,9 @@ def _resolve_spec(spec: dict): # step 3. replace fields 'modules' and 'versions' by respective lookups spec['modules'] = module_lookup spec['versions'] = version_lookup + # step 3a. add testcase lookup + spec['testcases'] = testcase_lookup + spec['tc_scripts'] = tc_script_lookup # step 4. resolve references # step 4a. resolve references to modules in includes # in this step, we also normalize the include form @@ -77,6 +88,11 @@ def _resolve_spec(spec: dict): {'module': module_lookup[inc['ref']], 'parameters': inc.get('parameters', {})} for inc in version['include'] ] + targets = defaultdict(set) + for inc in version['include']: + for target, tc_ids in inc['module'].get('targets', {}).items(): + targets[target].update(tc_ids) + version['targets'] = {target: sorted(tc_ids) for target, tc_ids in targets.items()} # step 4b. resolve references to versions in timeline # on second thought, let's not go there: it's a canonical extension map, and it should remain that way. # however, we still have to look for name errors @@ -148,106 +164,28 @@ def add_period(dt: datetime, period: str) -> datetime: raise RuntimeError(f'unknown period: {period}') -def parse_selector(selector_str: str) -> list[list[str]]: - # a selector is a list of terms, - # a term is a list of atoms, - # an atom is a string that optionally starts with "!" - return [term_str.strip().split('/') for term_str in selector_str.split()] - - -def test_atom(atom: str, tags: list[str]): - if atom.startswith("!"): - return atom[1:] not in tags - return atom in tags - - -def test_selector(selector: list[list[str]], tags: list[str]): - return all(any(test_atom(atom, tags) for atom in term) for term in selector) - - -def test_selectors(selectors: list[list[list[str]]], tags: list[str]): - return any(test_selector(selector, tags) for selector in selectors) - - -class TestSuite: - def __init__(self, name): - self.name = name - self.checks = [] - self.testcases = [] - self.ids = Counter() - self.partial = False - - def check_sanity(self): - # sanity check: ids must be unique - duplicates = [key for key, value in self.ids.items() if value > 1] - if duplicates: - logger.warning(f"duplicate ids in {self.name}: {', '.join(duplicates)}") - - def include_checks(self, module, parameters, sections=None): - missing_params = set(module.get('parameters', ())) - set(parameters) - if missing_params: - logger.warning(f"module {module['id']}: missing parameters {', '.join(missing_params)}") - return - self.checks.extend( - {**check, 'parameters': parameters} - for check in module.get('run', ()) - if sections is None or check.get('section') in sections - ) - - def include_testcases(self, testcases): - self.testcases.extend(testcases) - self.ids.update(testcase["id"] for testcase in testcases) - - def select(self, name, selectors): - suite = TestSuite(name) - if isinstance(selectors, str): - # convenience: allow callers to supply serialized form (they don't care, rightly so) - selectors = [parse_selector(sel_str) for sel_str in selectors.split(',')] - suite.include_testcases([tc for tc in self.testcases if test_selectors(selectors, tc['tags'])]) - return suite - - def eval_buckets(self, results) -> dict: - """ - returns buckets of test cases by means of a mapping - - None: list of missing testcases - -1: list of failed testcases - 0: list of aborted testcases - 1: list of passed testcases - """ - by_value = defaultdict(list) - for testcase in self.testcases: - value = results.get(testcase['id'], {}).get('result') - by_value[value].append(testcase) - return by_value - - def evaluate(self, results) -> int: - """returns overall result""" - return min([ - # here, we treat None (MISSING) as 0 (ABORT) - results.get(testcase['id'], {}).get('result') or 0 - for testcase in self.testcases - ], default=0) - - -def compile_suite(basename: str, include: list, sections: tuple = (), tests: re.Pattern = None) -> TestSuite: - suite = TestSuite(basename) - if sections: - suite.name += f" [sections: {', '.join(sections)}]" - suite.partial = True - if tests: - suite.name += f" [tests: '{tests.pattern}']" - suite.partial = True - for inc in include: - module = inc['module'] - # basic sanity - testcases = module.get('testcases', ()) - checks = module.get('run', ()) - if not testcases or not checks: - logger.info(f"module {module['id']} missing checks or test cases") - # always include all testcases (necessary for assessing partial results) - suite.include_testcases(testcases) - # only add checks if they contain desired testcases - if not tests or any(tests.match(ch['id']) for ch in testcases): - suite.include_checks(module, inc['parameters'], sections=sections) - return suite +def eval_buckets(results, testcase_ids) -> dict: + """ + returns buckets of test cases by means of a mapping + + None: list of missing testcases + -1: list of failed testcases + 0: list of aborted testcases + 1: list of passed testcases + """ + by_value = defaultdict(list) + for testcase_id in testcase_ids: + value = results.get(testcase_id, {}) + if isinstance(value, dict): + value = value.get('result') + by_value[value].append(testcase_id) + return by_value + + +def evaluate(results, testcase_ids) -> int: + """returns overall result""" + return min([ + # here, we treat None (MISSING) as 0 (ABORT) + results.get(testcase_id, {}).get('result') or 0 + for testcase_id in testcase_ids + ], default=0) diff --git a/compliance-monitor/monitor.py b/compliance-monitor/monitor.py index 7e7da62f5..4c1ccb2d4 100755 --- a/compliance-monitor/monitor.py +++ b/compliance-monitor/monitor.py @@ -49,14 +49,14 @@ try: - from scs_cert_lib import load_spec, annotate_validity, compile_suite, add_period + from scs_cert_lib import load_spec, annotate_validity, add_period, evaluate except ImportError: # the following course of action is not unproblematic because the Tests directory will be # mounted to the Docker instance, hence it's hard to tell what version we are gonna get; # however, unlike the reloading of the config, the import only happens once, and at that point # in time, both monitor.py and scs_cert_lib.py should come from the same git checkout import sys; sys.path.insert(0, os.path.abspath('../Tests')) # noqa: E702 - from scs_cert_lib import load_spec, annotate_validity, compile_suite, add_period + from scs_cert_lib import load_spec, annotate_validity, add_period, evaluate class Settings: @@ -138,7 +138,7 @@ class ViewType(Enum): templates_map = { k: None for k in REQUIRED_TEMPLATES } -_scopes = {} # map scope uuid to `PrecomputedScope` instance +_scopes = {} # map scope uuid to scope spec dict from YAML file class TimestampEncoder(json.JSONEncoder): @@ -239,96 +239,75 @@ def import_bootstrap(bootstrap_path, conn): conn.commit() -class PrecomputedVersion: - """Precompute all `TestSuite` instances necessary to evaluate the results of some version""" - def __init__(self, version): - self.name = version['version'] - self.suite = compile_suite(self.name, version['include']) - self.validity = version['validity'] - self.listed = bool(version['_explicit_validity']) - self.targets = { - tname: self.suite.select(tname, target_spec) - for tname, target_spec in version['targets'].items() - } - - def evaluate(self, scenario_results): - """evaluate the results for this version and return the canonical JSON output""" - target_results = {} - for tname, suite in self.targets.items(): - target_results[tname] = { - 'testcases': [testcase['id'] for testcase in suite.testcases], - 'result': suite.evaluate(scenario_results), - } - return { - 'testcases': {tc['id']: tc for tc in self.suite.testcases}, - 'results': scenario_results, - 'result': target_results['main']['result'], - 'targets': target_results, - 'validity': self.validity, +def _evaluate_version(version, scope_results): + """evaluate the results for `version` and return the canonical JSON output""" + target_results = { + tname: { + 'testcases': tc_ids, + 'result': evaluate(scope_results, tc_ids), } + for tname, tc_ids in version['targets'].items() + } + return { + 'result': target_results['main']['result'], + 'targets': target_results, + 'validity': version['validity'], + } -class PrecomputedScope: - """Precompute all `TestSuite` instances necessary to evaluate the results of some scope""" - def __init__(self, spec): - self.name = spec['name'] - self.spec = spec - self.versions = { - version['version']: PrecomputedVersion(version) - for version in spec['versions'].values() - } +def _evaluate_scope(spec, scope_results, include_drafts=False): + """evaluate the results for `scope` and return the canonical JSON output""" + testcases = spec['testcases'] + versions = spec['versions'] + version_results = { + vname: _evaluate_version(version, scope_results) + for vname, version in versions.items() + } + by_validity = defaultdict(list) + for vname, version in versions.items(): + by_validity[version['validity']].append(vname) + # go through worsening validity values until a passing version is found + relevant = [] + best_passed = None + for validity in ('effective', 'warn', 'deprecated'): + vnames = by_validity[validity] + relevant.extend(vnames) + if any(version_results[vname]['result'] == 1 for vname in vnames): + best_passed = validity + break + if include_drafts: + relevant.extend(by_validity['draft']) + passed = [vname for vname in relevant if version_results[vname]['result'] == 1] + return { + 'name': spec['name'], + 'testcases': testcases, + 'results': scope_results, + 'versions': version_results, + 'relevant': relevant, + 'passed': passed, + 'passed_str': ', '.join([ + vname + ASTERISK_LOOKUP[versions[vname]['validity']] + for vname in passed + ]), + 'best_passed': best_passed, + } - def evaluate(self, scope_results, include_drafts=False): - """evaluate the results for this scope and return the canonical JSON output""" - version_results = { - vname: self.versions[vname].evaluate(scenario_results) - for vname, scenario_results in scope_results.items() - } - by_validity = defaultdict(list) - for vname in scope_results: - by_validity[self.versions[vname].validity].append(vname) - # go through worsening validity values until a passing version is found - relevant = [] - best_passed = None - for validity in ('effective', 'warn', 'deprecated'): - vnames = by_validity[validity] - relevant.extend(vnames) - if any(version_results[vname]['result'] == 1 for vname in vnames): - best_passed = validity - break - if include_drafts: - relevant.extend(by_validity['draft']) - passed = [vname for vname in relevant if version_results[vname]['result'] == 1] - return { - 'name': self.name, - 'versions': version_results, - 'relevant': relevant, - 'passed': passed, - 'passed_str': ', '.join([ - vname + ASTERISK_LOOKUP[self.versions[vname].validity] - for vname in passed - ]), - 'best_passed': best_passed, - } - def update_lookup(self, target_dict): - """Create entries in a lookup mapping for each testcase that occurs in this scope. +def _update_lookup(spec, target_dict): + """Create entries in a lookup mapping for each testcase that occurs in this scope. - This mapping from triples (scope uuid, version name, testcase id) to testcase facilitates - evaluating result sets from database queries a great deal, because then just one lookup operation - tells us whether a result row can be associated with any known testcase, and if so, whether the - result is still valid (looking at the testcase's lifetime). + This mapping from pairs (scope uuid, testcase id) to testcase facilitates + evaluating result sets from database queries a great deal, because then just one lookup operation + tells us whether a result row can be associated with any known testcase, and if so, whether the + result is still valid (looking at the testcase's lifetime). - In the future, the mapping could even be simplified by deriving a unique id from each triple that - could then be stored (redundantly) in a dedicated database column, and the mapping could be from - just one id (instead of a triple) to testcase. - """ - scope_uuid = self.spec['uuid'] - for vname, precomputed_version in self.versions.items(): - listed = precomputed_version.listed - for testcase in precomputed_version.suite.testcases: - # put False if listed is False, else put testcase - target_dict[(scope_uuid, vname, testcase['id'])] = listed and testcase + In the future, the mapping could even be simplified by deriving a unique id from each pair that + could then be stored (redundantly) in a dedicated database column, and the mapping could be from + just one id (instead of a pair) to testcase. + """ + scope_uuid = spec['uuid'] + for tc_id, testcase in spec['testcases'].items(): + target_dict[(scope_uuid, tc_id)] = testcase def import_cert_yaml(yaml_path, target_dict): @@ -336,8 +315,8 @@ def import_cert_yaml(yaml_path, target_dict): with open(yaml_path, "r") as fileobj: spec = load_spec(yaml.load(fileobj.read())) annotate_validity(spec['timeline'], spec['versions'], date.today()) - target_dict[spec['uuid']] = precomputed_scope = PrecomputedScope(spec) - precomputed_scope.update_lookup(target_dict) + target_dict[spec['uuid']] = spec + _update_lookup(spec, target_dict) def import_cert_yaml_dir(yaml_path, target_dict): @@ -488,6 +467,16 @@ async def post_report( reportid = db_insert_report(cur, uuid, checked_at, subject, json_text) except UniqueViolation: raise HTTPException(status_code=409, detail="Conflict: report already present") + if 'versions' not in document: + # If this key is missing, this means we have a newer-style report that doesn't redundantly list + # results per version. One reason for this change is that the meaning of a testcase identifier + # no longer depends on the scope version, and we can quite simply read off the results from the + # invocations. -- Use the dummy version '*' as long as the db schema still expects a version. + document['versions'] = {'*': { + tc_id: {'result': result, 'invocation': inv_id} + for inv_id, invocation in document['run']['invocations'].items() + for tc_id, result in invocation['results'].items() + }} for version, vdata in document['versions'].items(): for check, rdata in vdata.items(): result = rdata['result'] @@ -504,24 +493,28 @@ def convert_result_rows_to_dict2( if grace_period_days: now -= timedelta(days=grace_period_days) # collect result per subject/scope/version - preliminary = defaultdict(lambda: defaultdict(lambda: defaultdict(dict))) # subject -> scope -> version + preliminary = defaultdict(lambda: defaultdict(dict)) # subject -> scope missing = set() - for subject, scope_uuid, version, testcase_id, result, checked_at, report_uuid in rows: - testcase = scopes_lookup.get((scope_uuid, version, testcase_id)) + for subject, scope_uuid, _, testcase_id, result, checked_at, report_uuid in rows: + testcase = scopes_lookup.get((scope_uuid, testcase_id)) if not testcase: # it can be False (testcase is known but version too old) or None (testcase not known) # only report the latter case if testcase is None: - missing.add((scope_uuid, version, testcase_id)) + missing.add((scope_uuid, testcase_id)) continue # drop value if too old lifetime = testcase.get('lifetime') # leave None if not present; to be handled by add_period if now >= add_period(checked_at, lifetime): continue - tc_result = dict(result=result, checked_at=checked_at) + # don't use outdated value (FIXME only necessary as long as version column still in db!) + tc_result = preliminary[subject][scope_uuid].get(testcase_id, {}) + if tc_result.get('checked_at', checked_at) > checked_at: + continue + tc_result.update(result=result, checked_at=checked_at) if include_report: tc_result.update(report=report_uuid) - preliminary[subject][scope_uuid][version][testcase_id] = tc_result + preliminary[subject][scope_uuid][testcase_id] = tc_result if missing: logger.warning('missing objects: ' + ', '.join(repr(x) for x in missing)) # make sure the requested subjects and scopes are present (facilitates writing jinja2 templates) @@ -530,7 +523,7 @@ def convert_result_rows_to_dict2( _ = preliminary[subject][scope] return { subject: { - scope_uuid: scopes_lookup[scope_uuid].evaluate(scope_result, include_drafts=include_drafts) + scope_uuid: _evaluate_scope(scopes_lookup[scope_uuid], scope_result, include_drafts=include_drafts) for scope_uuid, scope_result in subject_result.items() } for subject, subject_result in preliminary.items() @@ -561,7 +554,7 @@ def _build_report_url(base_url, report, *args, **kwargs): report_page = 'report_full' if kwargs.get('full') else 'report' url = f"{base_url}page/{report_page}/{report}" if len(args) == 2: # version, testcase_id --> add corresponding fragment specifier - url += f"#{args[0]}_{args[1]}" + url += f"#{args[1]}" # version no longer relevant return url diff --git a/compliance-monitor/templates/details.md.j2 b/compliance-monitor/templates/details.md.j2 index 332b11c23..dfaa9a0a4 100644 --- a/compliance-monitor/templates/details.md.j2 +++ b/compliance-monitor/templates/details.md.j2 @@ -28,8 +28,8 @@ No recent test results available. | testcase id | result | description | |---|---|---| {% for testcase_id in target_result.testcases -%} -{% set testcase = version_result.testcases[testcase_id] -%} -{% set res = version_result.results[testcase_id] if testcase_id in version_result.results else dict(result=0) -%} +{% set testcase = scope_result.testcases[testcase_id] -%} +{% set res = scope_result.results[testcase_id] if testcase_id in scope_result.results else dict(result=0) -%} | {% if res.result != 1 %}⚠️ {% endif %}{{ testcase.id }} | {#- #} {% if res.report -%} [{{ res.result | verdict_check }}]({{ report_url(res.report, version, testcase_id) }}) diff --git a/compliance-monitor/templates/report.md.j2 b/compliance-monitor/templates/report.md.j2 index e8dc9a3dc..5b27fabe4 100644 --- a/compliance-monitor/templates/report.md.j2 +++ b/compliance-monitor/templates/report.md.j2 @@ -4,33 +4,10 @@ - subject: {{ report.subject }} - scope: [{{ report.spec.name }}]({{ scope_url(report.spec.uuid) }}) - checked at: {{ report.checked_at }} - -## Results - -{% for version, version_results in report.versions.items() %}{% if version_results %} -### {{ version }} - -| test case | result | invocation | -|---|---|---| -{% for testcase_id, result_data in version_results.items() -%} -| {{ testcase_id }} {: #{{ version + '_' + testcase_id }} } | {{ result_data.result | verdict_check }} | [{{ result_data.invocation }}](#{{ result_data.invocation }}) | -{% endfor %} -{% endif %}{% endfor %} - -## Run - -### Variable assignment - -| key | value | -|---|---| -{% for key, value in report.run.assignment.items() -%} -| `{{ key }}` | `{{ value }}` | -{% endfor %} - -### Check tool invocations +- variable assignment: {% set comma = joiner(", ") %}{% for key, value in report.run.assignment.items() -%}{{comma()}}`{{ key }}`=`{{ value }}`{% endfor %} {% for invid, invdata in report.run.invocations.items() %} -#### Invocation {{invid}} {: #{{ invid }} } +## Invocation {{invid}} {: #{{ invid }} } - cmd: `{{ invdata.cmd }}` - rc: {{ invdata.rc }} @@ -45,6 +22,7 @@ - results {%- for resultid, result in invdata.results.items() %} - {{ resultid }}: {{ result | verdict_check }} + {: #{{ resultid }} } {%- endfor %} {% if invdata.stdout -%}