Add abilty to ignore request validation

[James Navin]

I would like to be able to simulate 'bad' requests in my Pacts, and validate that the correct 400 response is returned and that the shape of that response is correct.

Currently the validator treats the malformed request as a validation failure.

It would be great to be able to opt-out of request validation for specific interactions somehow. It would fine if this was done on a per-pact-file basis - I could just group all of my 'bad' requests into a one Pact.

[Ben Sayers]

Thanks for this @jnavin.

I have a concern with this request:

If we allowed you to ignore "bad" requests it opens up a can of worms where consumers and providers do not necessarily agree on what "valid" is. You can read more about this idea on this issue: https://bitbucket.org/atlassian/swagger-mock-validator/issues/64/strict-mode-to-make-warnings-errors

I'm concerned that if we don't have an answer on how to guarantee both sides agree on what valid is that we will cause more harm then good by allowing individual codebases to change the definition of valid. We've already run into situations where different teams using different versions of this tool became a problem where the only way to make one side green was the break the other side, this change feels like it will make this problem occur more often.

[James Navin]

I'm not sure I entirely follow the argument.

A large part of an API contract is what happens when things don't follow the happy path (e.g. what happens when I send invalid requests? what happens if my auth token is malformed? what happens if its missing entirely?). I think its reasonable for a consumer to want to simulate these scenarios in their tests, and want to verify that their assumptions about the shape of the responses from the provider are valid.

In a consumer-driven contract testing scenario these 'invalid' requests would just make up part of the set of expectations from the consumer (e.g. when I am missing a required field in a request I expect a 400 response that contains a body that looks like this).

As it stands, it is difficult to do this in the swagger-mock-validator as it will (rightly) fail the validation because the request shape is invalid.

If the concern is that consumer-provided Pacts with invalid requests will fail the provider builds, then perhaps some solutions could be:

• Allow you to mark the Pact itself as 'ignore request' and have the validator check for that flag during validation; or
• Put some smarts in the provider validation mode to detect validation errors in the request and expect non-2xx response codes in the response

[Ben Sayers]

I think I understand your use case and my concern is not with the use case itself, I agree we want to ensure our clients can handle these error cases. I'm more focused on making sure we don't allow consumers to fail provider builds.

Your idea to mark a pact interaction as "ignore request" would address my concern. We'd have to find a way to achieve that though, it seems to me that it would require a change to the pact specification and implementations?

I don't think I fully understand your second suggestion. I think what you are saying is when we validate a request and find it is invalid AND we find the response status code is 4xx we consider that valid?

I do see a workaround worth considering. You could make a valid request from the client but get the pact mock server to mock out a 4xx response. This would allow you to test the error handling behavior of the client while keeping the request/responses valid as far as swagger-mock-validator is concerned. However, if you are using the pact validation process in addition to the swagger-mock-validator this will likely create some complications for you when validating the pacts on the provider side using the pact validation, so worth keeping that in mind. Thoughts?

[James Navin]

Your idea to mark a pact interaction as "ignore request" would address my concern. We'd have to find a way to achieve that though

Its been a while since I looked at the Pact specification, but if it doesn't support an x-vendor extension mechanism, perhaps something as simple as a convention in the Pact name would work? (e.g. similar to smart commits or something where a directive is embedded in the Pact name - An invalid request example #ignore-request-validation)

I think what you are saying is when we validate a request and find it is invalid AND we find the response status code is 4xx we consider that valid?

Pretty much, yep. I don't really like this idea tbh, but in a RESTful API you would expect a syntactically invalid request to have a non-success 4xx response. You could bake this logic into the validator and convert request validation errors to WARN/INFO level IF the expected response included a 4xx status. Perhaps this could be an opt-in configuration?

I do see a workaround worth considering.

That would unblock consumer tests, true, but as you say it will make standard Pact validation more complicated. Also, in our specific use case we are using Pact to provide example interactions for our consumers alongside our Swagger spec. If we provide 'valid' requests but show an 'invalid' response it will likely cause confusion.