Conformance Tests

Issue #1199 new
Torsten Lodderstedt created an issue

OIDC4IDA is getting mature and being implemented in several places, we should talk about how we ensure interoperability among implementations.

Experiences in the context of OpenID Connect and FAPI have shown, automatic conformance testing is a reliable way to foster interoperability. I suggest we evaluate what it takes to build OIDC4IDA tests on top of the existing tests for OpenID Connect.

Comments (11)

  1. Joseph Heenan

    I can probably contribute to some extent to this discussion, I’ll try and remember to stay for the whole of next week’s call if it’ll be discussed there 🙂

  2. Torsten Lodderstedt reporter

    Next steps:

    • write up list of tests
    • identify existing implementations
    • use those implementations to test the test (must be accessible from the public internet)
    • funding and someone implementing the tests are required

    Assumption: will be built on the new Java-based OIDC conformance test suite

    Open: is there a specific security profile we assume? Seems we need to decide since the test driver (acting as RP) needs to send complete OIDC requests. There is a open ticket re complementary security profile (#1154).

  3. Joseph Heenan

    This was discussed a little more on today’s eKYC WG call, and I will attempt to summarise it as:

    Nat expressed that there is reputational risk associated with allowing the certification of implementations with weak security, and as such it was expressed that implementations intending to demonstrate their conformance to eKYC would be required to support a FAPI security profile. Mark/Joseph agreed and no one dissented.

  4. Torsten Lodderstedt reporter

    Idea discussed today: we start with tests of request and corresponding respond payload and leave aside the protocol flow.

  5. Joseph Heenan

    Discussed again today. Unfortunately I didn’t get the chance to raise the above idea with the certification team, but will endeavour to do so on the next certification call (4th Jan I think).

    Things mentioned today:

    1. The spec generally only contains simple examples, but the schemas allow much more complex requests, do we expect certification to make more complex requests?
    2. We presumably need to come up with some sets of precanned requests that we expect people to execute, grouped by the claim name, with the intention people would run the test sets for the claims they list in claims_in_verified_claims_supportedin their discovery document.
  6. Torsten Lodderstedt reporter

    re 1) we (yes.com) can contribute more complex test cases.

    re 2) I think the request and response payload needs to be precanned but variable, meaning the concrete trust frameworks, for example, nee to be determined from the OP’s metadata.

  7. Mark Haine

    In the eKYC conformance focus group we discussed finding the answers to the following questions…

    • What should we test?
    • How should we test?
    • How will we make allowance for the variations likely in the security profile?

  8. Log in to comment