Flaky JWKSource tests fail intermittently

[Vladimir Dzhuvinov]

JWKSourceUnknownKeysTest.testMultiThreadedKnownAndUnknownKeys:212 expected:<8> but was:<10>

With a Java 17, a lot.

[Vladimir Dzhuvinov]

JWKSourceKeyRotationTest.testMultiThreadedKeyRotation:155 expected:<9> but was:<10>

With the configured Bitbucket pipeline

[Vladimir Dzhuvinov]

JWKSourceUnknownKeysTest.testMultiThreadedKnownAndUnknownKeys:212 expected:<8> but was:<11>

With the configured Bitbucket pipeline

[Thomas Rørvik Skjølberg]

Perhaps the Thread.sleep(1000) in runForDuration(..) is a bit long when the cache timings are 2 or 3 seconds?

[Thomas Rørvik Skjølberg]

If the tests just run for too long, perhaps because starting all the threads take more time on the CI server, then it might be an idea to time the actual run time and adjust the assertEquals.

[Yavor Vasilev]

Merged PR on behalf, we have a green light here:

https://bitbucket.org/connect2id/nimbus-jose-jwt/pipelines/results/190

[Thomas Rørvik Skjølberg]

At least it did not get any worse
Were you able to trigger the build multiple times (to check if still flaky)?

[Yavor Vasilev]

Red light from a run just now:

https://bitbucket.org/connect2id/nimbus-jose-jwt/pipelines/results/191

Is it a good idea to make the test run in loop several times and break when it passes?

[Thomas Rørvik Skjølberg]

Sure, that is a good plan. The main point of the tests is not the run time itself, but that multithreading works as expected.

[Thomas Rørvik Skjølberg]

Try again? https://bitbucket.org/connect2id/nimbus-jose-jwt/pull-requests/101

[Thomas Rørvik Skjølberg]

Still fails.

[Thomas Rørvik Skjølberg]

Are you able to trigger the build repeatedly? Now one test fails, would be interesting to see whether both still fail.