Certification/conformance: Strictness of checking error responses

[Joseph Heenan]

An ecosystem has queried the certification team as to whether particular conformance tests are overly strict and perhaps not adding significant value.

The particular issue in this case is when the FAPI2 tests attempt to do an authorization using response_type=code id_token (which is not permitted in FAPI2). One of the accepted outcomes here is an error from the PAR endpoint, where https://www.rfc-editor.org/rfc/rfc9126.html#section-2.3 seems to be the relevant place to start and says:

The authorization server returns an error response with the same format as is specified for error responses from the token endpoint in Section 5.2 of [RFC6749] using the appropriate error code from therein or from Section 4.1.2.1 of [RFC6749].

and https://www.rfc-editor.org/rfc/rfc6749#section-5.2 seems clear to me than an 'unauthorized_client' error (i.e. “The client is not authorized to request an authorization code using this method”.) is sent with a 400 http status code. The system in question is returning a 401 response instead, which causes a failure in the test and prevents certification.

As in this case, I don’t think we would ever expect the client to have logic like “if try response_type code id_token and I get a http 400 with JSON that has an unauthorized_client error then try response code instead”, so potentially allowing any 4xx error is sufficient to achieve the underlying goal that the client is not permitted to use that response type (or all least downgrading the failure to a warning).

In this working group agreed this might be a suitable approach to be used across all FAPI conformance tests, we’d probably need a bit more guidance as to what kind of errors fail into this category and which ones we do have an expectation that at least some clients might react to and recover from automatically.

(I thought we’d discussed this topic before, but I can’t find a prior issue. #434 is vaguely related, but about user visible error messages.)

[Filip Skokan]

Response status codes as they are defined and re-used to mean pretty much anything in oauth2 (especially 400) are hardly ever useful for code flow logic.

But in the past there was no leeway on the suite’s side for different status codes than the ones defined and existing products and software had to adjust to get a certified stamp, despite offering meaningful status codes, they had to backtrack to overloaded status codes.

If the suite makes one exception, more requests will come.

[Joseph Heenan]

Thanks Filip for your thoughts Filip!

We discussed this yesterday on the WG call (including your feedback). I don’t think we ever really considered asking the working group about the value of these tests before. [and I have no idea why the specs are so opinionated about the status codes returned, but even recent specs like PAR seem to have inherited that.]

There did seem to be some agreement that there wasn’t much value in being strict in these cases, and your point that it may become a slippery slope was discussed.

My concrete suggestion at this point is that we could change tests like this to expect a 4xx (and fail if it isn’t a 4xx), but only warn if it’s not the expected 400. I think there were two possible policies suggested:

1. any error that’s considered fatal, i.e. there’s no expectation the client could take some action automatically and go ahead to make a successful request, or
2. any test that currently allows the flow to end at the authorization endpoint with an error shown to the user (which is a subset of '1')

‌

[Joseph Heenan]

This was discussed further on today’s call with a consensus towards following the more restrictive option ‘2' (only applying a loser policy where the test could fail at the authorization endpoint with an error displayed to the user) for now until we have some concrete examples of '1’ to discuss.

[Dave Tonge]

Changes made to test suite

[Joseph Heenan]

The certification team will make a change for https://gitlab.com/openid/conformance-suite/-/issues/1227 which falls inside the more restrictive ‘2' option that the WG were happy for us to change. We don’t plan to generally review the tests at this stage and may make further changes if people raise issues with the certification team.