Optimize --find-links=<path-to-dir>

[notatallshaw]

Fixes #12320 by implementing the optimization strategy:

• All instances of _FlatDirectorySource only scan the relevant path once (instead of currently which is 2*(number of projects pip considers installing)) by caching the result of scanning

Further optimizations were added:

• Remove Path construction for only interating directory, as the result was just stingified anyway
• _FlatDirectorySource only considers file names in the path that have a valid sdist of wheel name
• _FlatDirectorySource only considers file names that are relevant to the current project Pip is considering, which produces a huge speed up where there are a lot of projects to be installed

To make sure this optimization did not produce unintended slowdowns I tried 4 different requirements on my machine using the steps outlined in #12320, taking the median of each run I saw all 4 saw performance improvements (although for a small number of wheels it was small enough to be negligible):

Requirement	Pip Main Median Time	Pip Optimized Median Time	Relative Performance Improvement
requests	0.3 seconds	0.296 seconds	1.3%
pandas[performance,excel,html,parquet]	0.662s seconds	0.645s seconds	2.6%
apache-airflow[all]	45.633 seconds	30.724 seconds	32.7%
homeassistant from #12320	159.206 seconds	68.969 seconds	56.7%

[notatallshaw]

Optimizations strategies in this PR might be able to be applied to other ways Pip collects links. I will take a look at them when I have a way to consistently benchmarking those.

[notatallshaw]

Profiling the homeassistant use case shows that in Pip main 47% of the time was taken while under the function pip._internal.index.package_finder.find_best_candidate, under this optimized branch that is reduced to 0.66% of the time.

The biggest hotspot under this branch for that use case is now pip._vendor.resolvelib.resolvers._is_current_pin_satisfying which is being called millions of times by pip._vendor.resolvelib.resolvers.resolve, if this can somehow be optimized it would have performance gains for almost all Pip installs.

[notatallshaw]

Updated to significantly simplify the new class _FlatDirectoryToUrls, rather than it implementing some singleton-like behavior in _FlatDirectoryToUrls the class _FlatDirectorySource just keeps a class level variable keeping track of what _FlatDirectoryToUrls classes have already been instantiated and can be reused.

Also cleaned the commit history so it's just 3 commits.

[pfmoore]

I do intend to try to review this at some point, but life has been hectic, so I can't promise when I'll be able to get to it.

[notatallshaw]

I do intend to try to review this at some point, but life has been hectic, so I can't promise when I'll be able to get to it.

Absolutely no rush, I don't consider this PR particularly critical in of itself.

But I planned to benchmark Pip's other ways of collecting packages and seeing if I can reuse the same approaches to improve their performances as well. And it also it helps surface other O(n^2) issues while profiling Pip collecting lots of packages.

[notatallshaw]

Thanks for merging, I will keep an eye out on the issue tracker after the next Pip release, this will probably have an unintended effect on someone's unusual workflow.