I do agree with your instinct that tests need not be part of the distribution, but since they are, we should remain considerate of the implications of changing it.
Nevertheless, I'll accept the change and make a backward-incompatible release.
Thank you, you made the right call. In my opinion, though, the tests are not part of the public API and thus any code that relies on those test fixtures is broken anyway. I consider only changes in the published API to be backwards incompatible.