Let schema validation during CI fail when attributes are present which are not defined

[Kai Lehmann]

This is not related to the draft content itself, but the CI process. As we are missing updating the JSON schema quite often, we could try to configure the CI process to fail schema validation of updated or new examples whenever there are attributes or enum values used which are not defined within the schema. This can help us to prevent forgetting schema updates.

[Kai Lehmann]

I have been looking into this and it seems there is no obvious option in the jsonschema library we use to validate the JSON files to add the additionalPropertiesattribute to each schema object definition. It might be possible to code a custom validator and add it to the schema validation environment though.

[Daniel Fett]

I created this PR to introduce a “strict” testing mode: https://bitbucket.org/openid/ekyc-ida/pull-requests/161

However, the tests currently fail, because we sometimes intentionally make use of non-standardized attributes.

[Daniel Fett]

Some errors occur because we currently don’t define the OIDC Core claims that can go directly in the “userinfo” or “id_token” part of a request.

FAILED tests/test_schema_application_strict.py::test_request_schema[simple_id_token.json] - jsonschema.exceptions.ValidationError: Additional properties are not allowed ('given_name' was unexpected)
FAILED tests/test_schema_application_strict.py::test_request_schema[id_token.json] - jsonschema.exceptions.ValidationError: Additional properties are not allowed ('email', 'picture', 'preferred_username' were unexpected)
FAILED tests/test_schema_application_strict.py::test_request_schema[id_token_id_document.json] - jsonschema.exceptions.ValidationError: Additional properties are not allowed ('email', 'picture', 'preferred_username' were unexpected)

Can anybody fix that? It was not immediately clear to me how to include the elements already defined in “claims_type” here.

[Kai Lehmann]

Shouldn’t the schemas we provide only be applied to the verified_claims part of the examples? So for instance when we want to validate the simple_id_token.json example, the id_token element contains the verified_claims element and the schema validation should be bound to the verified_claims element only and no validation should be performed on the surrounding element.

that said, we want to verify that the property is actually named “verified_claims”. So I think we could allow additional properties in an object which contains the “verified_claims” property and do not set “additionalProperties=False in the python test whenever an additionalProperties attribute is already defined in the schema.

I modified the PR accordingly and now the tests are green. I’m not sure if there are any unwanted side effects though. It could be that others might use/reference the provided schemas to validate the entire JSON documents and they would like to disallow additionalProperties in the surrounding objects while our schema definition allows them now.

The best approach would be to select the verified_claims element and restrict validation on that sub element only.

[Daniel Fett]

Thanks Kai! That makes a lot of sense and we just merged the PR.

[Daniel Fett]

PR #161 merged.