Convert test runner to use pypiserver package as standalone process

[matteius]

• Converts the pipenv test runner to use pypiserver.
• Invokes pypiserver correctly for all supported OS types
• Uses waitress for Windows only because it reduced the run time of windows test runner by 10 minutes from the default wsgi bottle.
• Does not use waitress for Ubuntu because it slowed it down significantly, nor for mac OS which showed no difference using it.
• Refactor test naming and cleanup conftest file to be more certain about temp directory cleanup
• Tableflip on the dep_to_pip style tests and realize that there is still some cross tests contamination going on that sometimes confused requirementslib into adding a specifier for requests without a specifier -- I ultimately concluded that I don't have time currently to figure out how to prevent this other than to convert those requests and requests with extras pip_to_deps tests to use uvicorn as its test subject instead, since it too has extras and we don't use it in other tests that pull in the vcs setup.py metadata artifacts. We can see on the latest main build this failed again too, so we basically have been getting lucky and this change to uvicorn at least gurantees the build won't be flaky anymore for this particular reason: https://github.com/pypa/pipenv/runs/8179118655?check_suite_focus=true
• Updated dev documentation and the scripts for invoking pytest to startup the pypiserver ahead of time.
• I did some speed analysis and I feel this is on par with our current test runner, but is way better because we eliminate the fixtures that invoke several threads of an old unsupported pypiserver and instead the tests are CPU lighter now when they run even if they take a similar amount of time.
• This really opens the door to better debugging because it becomes easy to access the same private pypi server that the tests are referencing and check its behavior and the available packages. The options we can pass to this pypiserver are way more expansive, and likely to improve overtime as that project has a new maintainer that appears interested in accepting contributions. For example, right now there is no way to specify thread count to the bottle wsgi server, and so for waitress it defaults to 4, but the test runner has 2 cores so this is overkill -- we can probably get in the future an ability to specify this value to experiment with runtimes.
• We are closer than ever with this PR to being able to support private pypi tests that involve credentials 🎉
• Probably other benefits to doing this.

Deferred for Future

• Extends the hash lookup method to consider if the index being used supports the same pypi API for retrieving package hashes. -- Deferred to future PR since I shouldn't add a functional change to pipenv in this re-work of the testing framework.
• Associated PR for pypiserver and when I subbed out feature it shaved almost a 30-60 seconds off my test runner locally: Prototype supporting the same package hash json API response that pypi provides pypiserver/pypiserver#440

Artifacts and Stats

First run where this PR refactor was really successful:

Second run where I pushed doc/news changes and let it re-run:

###Latest main branch run (for possible comparison)

The checklist

• A news fragment in the news/ directory to describe this fix with the extension .bugfix.rst, .feature.rst, .behavior.rst, .doc.rst. .vendor.rst. or .trivial.rst (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

[matteius]

These tests are the black magic of the devil and old setuptools -- how they ever passed I have no clue and vote we should drop them now. Just read the comments, it feels like a very old problem it was checking for, and with time we would drop these tests anyway.

[oz123]

I believe, though I haven't tested it, that adding gunicorn on Ubuntu and Mac might improve the speed of the tests.
WSGIRef is darn slow, especially when running in parallel

[matteius]

I believe, though I haven't tested it, that adding gunicorn on Ubuntu and Mac might improve the speed of the tests. WSGIRef is darn slow, especially when running in parallel

It might have helped ubuntu but I think it hurt Mac OS run time so I am going to change gunicorn to just be installed for linux.