For installation of test packages it would be great if wheels could be used instead.
This is difficult to work for all possible python versions but to start with, universal wheels could be supported.
In this case the universal wheel could be produced in the first step. Then tox can use the test_requires and install_requires to build the deps used in the testenv.
If you want to support multi-python versions at build time, tox could read the setup.cfg wheel config file using the wheel module and drop down to using sdist for those builds.