Conformance Tests

[Torsten Lodderstedt]

OIDC4IDA is getting mature and being implemented in several places, we should talk about how we ensure interoperability among implementations.

Experiences in the context of OpenID Connect and FAPI have shown, automatic conformance testing is a reliable way to foster interoperability. I suggest we evaluate what it takes to build OIDC4IDA tests on top of the existing tests for OpenID Connect.

[Joseph Heenan]

I can probably contribute to some extent to this discussion, I’ll try and remember to stay for the whole of next week’s call if it’ll be discussed there

[Torsten Lodderstedt]

Next steps:

• write up list of tests
• identify existing implementations
• use those implementations to test the test (must be accessible from the public internet)
• funding and someone implementing the tests are required

Assumption: will be built on the new Java-based OIDC conformance test suite

Open: is there a specific security profile we assume? Seems we need to decide since the test driver (acting as RP) needs to send complete OIDC requests. There is a open ticket re complementary security profile (#1154).

[Joseph Heenan]

This was discussed a little more on today’s eKYC WG call, and I will attempt to summarise it as:

Nat expressed that there is reputational risk associated with allowing the certification of implementations with weak security, and as such it was expressed that implementations intending to demonstrate their conformance to eKYC would be required to support a FAPI security profile. Mark/Joseph agreed and no one dissented.

[Torsten Lodderstedt]

Idea discussed today: we start with tests of request and corresponding respond payload and leave aside the protocol flow.

[Joseph Heenan]

Discussed again today. Unfortunately I didn’t get the chance to raise the above idea with the certification team, but will endeavour to do so on the next certification call (4th Jan I think).

Things mentioned today:

1. The spec generally only contains simple examples, but the schemas allow much more complex requests, do we expect certification to make more complex requests?
2. We presumably need to come up with some sets of precanned requests that we expect people to execute, grouped by the claim name, with the intention people would run the test sets for the claims they list in claims_in_verified_claims_supportedin their discovery document.

[Torsten Lodderstedt]

re 1) we (yes.com) can contribute more complex test cases.

re 2) I think the request and response payload needs to be precanned but variable, meaning the concrete trust frameworks, for example, nee to be determined from the OP’s metadata.

[Mark Haine]

In the eKYC conformance focus group we discussed finding the answers to the following questions…

• What should we test?
• How should we test?
• How will we make allowance for the variations likely in the security profile?

‌

[Daniel Fett]

Our working document: https://docs.google.com/document/d/1SX2_SjcMUQJ6SQEuNrhNjqAqpQjTUqkHl_qCReSv9-Y/edit?usp=sharing

[Mark Haine]

Just noting a new challenge we need to address in the context of conformance testing of OpenID Connect for IDA.

Versioning of testing and linkage to versioning of draft(s).

At present there is a beta conformance testing too that has been based largely upon Implementers draft 3 but the draft has developed since that beta test conformance tool was delivered.

[Mark Haine]

There was discussion in today’s working group about versioning of draft and of conformance/certification and the outcome was pretty much…

• Conformance beta without certification would be valuable if delivered in May timeframe
• This wil coincide with TISA pilot and they will likely deliver 4 IDPs to consume the OIDC 4 IDA conformance testing tool
• The version to be delivered in May should be the draft spec as it was in Bitbucket source after commit 4c67116 which includes the changes merged in pull request #87 (assurance process) and pull request #106 (addition of claims schema)

@Joseph Heenan let me know if I missed anything

‌

[Joseph Heenan]

That sounds right, thanks. From a certification team perspective we’re essentially now waiting for the TISA project to provide us with access to a compliant server to test the tests against.

The implication of using a spec from a particular git commit is that we won’t be aligned to an implementers draft, and hence launching a formal certification program will have to wait until the next implementers draft appears and we’ve re-aligned the tests to that.

‌