Strict mode to make warnings errors

[Jaiden Ashmore]

Problem

I have a pact.json that includes a query parameter that does not exist in the swagger.json for the provider resulting in a warning: spv.request.query.unknown but a successful validation. Instead I would prefer it to fail because it is more likely that the developer has not done what is intended and this should not be merged into master.

Potential solution

Add a strict cli parameter

swagger-mock-validator --strict swagger.json pact.json

[Ben Sayers]

We've thought about adding the ability to configure the validator. Actually what we had in mind was something more general where each validation rule could be set to "ignore", "warning" or "error", with each rule having a default value. What you are asking for is essentially a second set of defaults that is stricter then the current set of defaults, but I can imagine a different team wanting a different definition of "strict" so perhaps making the configuration more general sets us up better for the future.

What has stopped us from making these types of changes in the past is concern that this opens the door to consumers and providers configuring the tool differently. When this occurs there is no longer agreement between the consumer and provider on what "valid" means. The risk is that one side can make a change that the other does not consider valid and breaks their build. For example say Jira Frontend runs the validation without the strict flag and publishes a Pact file containing an undefined query parameter, but when Jira Backend runs the validation they use the strict flag. Now Jira Backend's build is broken.

There are several ways we can resolve this concern:

Change the default for everyone to disallow unknown request query params

The reason we didn't do this in the first place is it becomes a blocker for fast 5 style changes where the consumer deploys first. For example if we wanted to add a new query param to an API in a world where unknown query params were disallowed then the provider has to change the Swagger file as the first step. Our thinking was that in general more teams would prefer the flexibility to deploy in any order over the strictness.

We also have been operating on a general principal that this validator should not fail unless we are certain it will not work in production. In this case additional query parameters will not cause a failure.

Share the validation rule configuration

If a consumer or provider had the ability to publish the validation rule configuration to a shared location for a specific integration and the validator had the ability to retrieve this configuration then both sides will agree on what is considered valid. Now we have started work on the contract testing cli and the contract testing broker this is a feasible option as we will soon have the infrastructure in place to make this a seamless process.

Please add any other thoughts you have on this and we can figure out where to go with this feature request.

[Jaiden Ashmore]

Hmmm interesting, this is a lot of scenarios that I had not considered when I was reporting this.

For example say Jira Frontend runs the validation without the strict flag and publishes a Pact file containing an undefined query parameter, but when Jira Backend runs the validation they use the strict flag. Now Jira Backend's build is broken.

Yes this is definitely not what we want, so both sides of the contract would definitely need to have the same levels of strictness.

The reason we didn't do this in the first place is it becomes a blocker for fast 5 style changes where the consumer deploys first.

In my case, I am making both the backend and frontend changes and I would just wait until the backend changes have been deployed before merging my frontend changes. It would be no problem to merge them at the same time and eventually it will begin working (or not considering I could have a typo in the query parameter name on one side or another).

I am more wanting to protect against this scenario where on both sides the unit tests and functional tests pass but the their usage are not correct in that they have passed in a completely incorrect query parameter and ultimately the feature does not work. I guess it is up to the developers to be confirm there changes and follow up with the warnings to go away?

[Ben Sayers]

This feature request opens a can of worms regarding consumers and providers agreeing on what things are considered valid vs invalid. We do not have any ideas on how to resolve this concern and we feel that overall users of swagger-mock-validator will have worse problems compared to today if we were to add this feature without addressing this concern. As a result I will close this as won't fix.

I'm happy to reconsider if new ideas on how to address the concern come to light. If you have any please let us know!