Support for publishing the provider verification results back to pact broker

[Sanjit Mohanty]

Hi, Is there a support where the provider validation results could be published back to the pact broker?

This is quite necessary to close the contract validation loop as this brings more visibility when the provider is able to publish the validation results back to the pact broker.

Regards, Sanjit

[Ben Sayers]

Thanks @sanjitmohanty, this isn’t a feature we currently support so I added this request to our backlog. Just to help you with your own planning please do not expect this to be implemented anytime soon as the team is very busy.

[tissot mickael]

Is there some news about this topics ?

I meet the same issue : on my provider side, i can easily use PactProviderValidator to verify my contract from a pact broker.
But, the test doesn’t generate any report so no possibility to send it even in custom way to the broker.
More over the pact maven plugin just try to run the tests against a live instance of my provider and there is no way to customize it to make it run the test with the swagger PactProviderValidator.

Even with this maven plugin i have no possibility to send report to the broker.

I have two choices now from a provider perspective :

• run the tests with your PactProviderValidator library but could not have report in broker, then have a lake of functionality in my CI CD pipeline, because the validation matrix, and the “can i use” method will never reflect the reality
• run the test without your PactProviderValidator library, and make the execution longer, harder to maintain, deal with integration (even if i mock all the dependencies of my provider, i have to deal with data management …)

I’d really like to deal with solution number one, and it would be so great if we could push the result back into the broker.

By the way, is it possible to know how you achieve this at Atlassian ?

Thank you

‌

[Ben Sayers]

@tissot mickael No there hasn’t really been any developments in this space.

Since you mentioned this codebase being a maven plugin, I wonder if you’ve confused this nodejs/cli tool with the java library swagger-request-validator? We chose similar enough names that it’s easy to confuse them so sorry about that!

Inside Atlassian we have a wrapper tool that is responsible for orchestrating the validation and sharing of contracts to make it easy for teams to implement a closed contract loop between all consumers and providers. For validation it uses this library, as well as openapi-diff and a few others. We’ve thought about open sourcing this tool, however it currently has a dependency on an internal service where we publish the pact and swagger files. We were hoping that the Pact Broker would be updated to allow Swagger files to be published to it, this would allow us to add Pact Broker support into our tool and then open source it. However, my understanding is the owners of the Pact Broker project decided to make publishing Swagger files a paid feature.

We briefly discussed how we use Swagger Mock Validator on the consumer and provider side in a talk we did a while back if you’d like to know more:
https://youtu.be/-6x6XBDf9sQ?t=2126 (starting at 35:26)

[Abhi Nandan]

Has there been any progress on this? This tool is really useful and saves a lot of testing effort by reusing openapi3 specification; however, it doesn’t close the loop with the provider results published on pact broker, which restricts us from using Pact Broker to its full potential. Is there any alternate way to get the complete visualisation of provider-consumer interactions & compatibility? It would be great if you share some insights on this.
Thank you

[Renan Carlos Rodrigues]

@Abhi Nandan I’ve forked this repo and implemented a --publishResults option that may help you.
Follow: https://bitbucket.org/rcrodrigueszup/swagger-pact-validator/src/0dd9135674a5fc503b2e3cadad0fa1cb06a1d5f5/?at=feat%2Fpublish-results

[Renan Carlos Rodrigues]

@{557057:526b7077-83d4-4cfb-955a-e261d129103c} @Ben Sayers would this be a good candidate for a PR ?

[Bruno Mendola]

@Renan Carlos Rodrigues I vote for the PR, please

I came here looking just for this feature and it would be really useful!

Thank you!

‌

[Bruno Mendola]

Hi @Renan Carlos Rodrigues , could you please submit a pull request for the feat/publish-results branch?

It’s a very useful feature and it would be awesome to have it integrated in the official project.

Thank you!

[Renan Carlos Rodrigues]

@Bruno Mendola I’m getting a 403 response when trying to access new PR page. I’m not a very experienced user of bitbucket tho.
@{557057:526b7077-83d4-4cfb-955a-e261d129103c} Any ideas on this ?

[Bruno Mendola]

Hi @{557057:526b7077-83d4-4cfb-955a-e261d129103c} , does this project accept pull requests?

How can @Renan Carlos Rodrigues submit the new feature he implemented?

It would be nice to have it on the original project, instead of maintaining a fork.

Thank you!

[Tim Jones]

I’m curious about this feature - what is the provider whose verification status is published? I don’t think the spec is a provider, and I don’t think it would be correct for a spec-implementing provider to be considered verified by swagger-mock-validator without at least some extra testing.

If you have a pact file generated from a consumer pact test, then each interaction in that pact says "I can send X request. When the server is in state S, this should result in Y response, which I can also understand"

Checking the pact file against the swagger says "are request X and response Y allowed by the spec"

Verifying the pact file against the actual provider says "when the server is in state S and receives X request it generates Y response".

I think these are subtly (but significantly) different. A big advantage of a pact contract over a swagger spec is the request and response pairing - something that is missing entirely in spec-only validation. Even in the case where this doesn’t matter (eg there is only one valid response type from each endpoint, including no error cases), then I don’t think it’s appropriate to validate the pact against the spec and consider the provider verified - what if the provider doesn’t implement the spec properly? How would you validate that?

What use cases do people have for publishing results? Are people using it in lieu of pact verification? Or is it part of a more complex tool chain where the spec is also verified against the provider (I think the simplest use case might be where the spec is generated from the provider implementation, so you know it matches if the generator was correct).

Even if that’s happening, you still lose the request/response pairing. Treating a spec validation as provider verification feels risky to me- I’m keen to hear about how people would like to use it.

[Bruno Mendola]

In my case we have contract tests, then e2e tests.

I’d use this new feature in swagger-mock-validator to not have to implement the pact broker API client myself, since it’s already something that’s well done here.

For me, it makes sense that after the pact publishing you also provide a function to publish the verification result.

[Renan Carlos Rodrigues]

@Tim Jones I totally agree with you.

‌

I’m relying on these premises when using this feature:

• In order to test the Pact file against the Spec, it’s a MUST to guarantee that the provider implements the spec thoroughly. (i.e using Codegen)
• I do believe that giving a green light on the Pact Broker could lead to misinterpretation on the actual contract health status. Me and my team are aware of that and we do use this feedback loop’s output carefully. Setting a SPEC flag, or TAG, on the Pact Broker could help preventing this.

I’m not saying this is the best way to make sure the any contract is healthy, but I do believe it’s better than nothing in scenarios where one of your project’s providers is not able to implement the PACT validation on his side, but he does maintain a spec.

‌

[Sebastian Tello]

Very late response, I’ve been working on other projects

• Regarding PR permissions, I’ve made some config changes, not sure if they are allowed now
• As for Tim’s comment (I can’t seem to be able to find the right user when @ mentioning), yes, this tool works on the assumption that the swagger/openapi specification handed out by the provider is in sync with the actual implementation.. this is a responsibility of the provider and validation of that assumption is outside the scope of this tool, the methods to validate that may vary depending on the tech stack (e.g. generate server code out of a hand-written swagger file, or a swagger file out of the server code/annotations, or validate through tests + client/server middleware, or tools such as dredd)
• IMO, I don’t see much value in pairing request-responses.. as for contract testing we want to validate that the requests your client generates are valid and that your client can handle responses properly.. verifying that request X results in response Y falls into functional testing territory and IMO it is a responsibility of provider to be verified at another level of testing (unit/integration)

‌

[Tim Jones]

as for contract testing we want to validate that the requests your client generates are valid 

Perhaps we have different definitions of contract testing, but this feels like schema testing to me.

I think a contract test verifies that your consumer is capable of communicating with the provider (and vice versa). Validating the contract against a particular spec doesn’t actually achieve this - a message may be syntactically correct but meaningless in the context of the communication (eg Noam Chomsky’s “colourless green ideas sleep furiously“ example - syntactically correct, but not very meaningful).

Matt Fellows wrote about this here https://pactflow.io/blog/schemas-are-not-contracts/ , and we also spoke about it in an interview here https://www.youtube.com/watch?v=O_lXopbrq_Y

verifying that request X results in response Y falls into functional testing territory

I agree that you shouldn’t be doing functional testing with your contract tests, but I don’t agree with this sentence. In a contract test, what is under test is the ability of the client to send X request and understand Y response. What is not under test is the functionality behind all the possible reasons the server might generate Y. You’re testing the communication pieces. Maybe there’s a structural mistake where the server can marshal that particular response, but not for that particular request - and if there is, a contract test should pick it up (without doing functional testing).

Of course, It’s ok from a risk-reduction perspective for the test to have some accidental functional coverage - but I wouldn’t go out of my way to make that happen.

‌