Tooling for automated stats gathering

[brandtbucher]

We have a wonderful mechanism (--with-pystats) for quantifying the success of many of our optimizations. However, there is currently quite a bit of friction involved in collecting that data. I think that the situation can be improved without too much difficulty.

My current wishlist regarding pyperformance runs built --with-pystats:

• A cron job to run pyperformance with stats turned on, perhaps weekly, and check the results into the ideas repo. This can run on GitHub actions, since we don't actually care about performance stability. It can also be parallelized, and maybe even make use of Pyperformance's --fast option.
  • Bonus points: compare the stats with the previous run, and surface any "interesting" regressions (there have been times where hit rates have plummeted without our knowledge before).
• A way to run a stats build of pyperformance using a label on any CPython PR, and report the results in a comment. Currently, collecting stats for a PR is a slow process that must be completed locally.
  • Bonus points: also run stats for the base commit.
    • Bonus bonus points: compare the stats automatically, surfacing any "interesting" changes.

I also know that @markshannon has also expressed a desire to have pyperformance (or maybe pyperf?) turn stats gathering on and off using 3.12's new sys utilities before and after running each benchmark, so that we're gathering just stats on the benchmarks themselves, and ignoring the external benchmarking machinery.

CC @mdboom

Individual tasks to get there:

• Prototype
• pyperf only collect stats during benchmark code
• Update the summarize_stats.py script to support comparing results
• Make a release of pyperf
• Upgrade the pyperf requirement in pyperformance
• Make a release of pyperformance
• Add periodic Github Action to ideas repo
• Add PR Github Action to python/cpython

[mdboom]

I think this is a great idea. Unlike timing benchmarks, this is a lot more deterministic (though I suspect not 100% deterministic). I like the idea of stats helping to be a bit of a "tie breaker" when maybe the raw timings are too close to call (only for PR's that are obviously trying to improve the specializer, of course).

I also know that @markshannon has also expressed a desire to have pyperformance (or maybe pyperf?) turn stats gathering on and off using 3.12's new sys utilities before and after running each benchmark, so that we're gathering just stats on the benchmarks themselves, and ignoring the external benchmarking machinery.

Yeah -- I was just working on this part this morning: psf/pyperf#140

[mdboom]

Adding this automation to PRs will require committing more automation to upstream cpython, where there is (understandable) wariness to adding more complexity. As a half-way measure, it turned out to be really easy to add support for running on an arbitrary fork and branch to the automation in the ideas repo:

I think that might be good enough for now, and it at least addresses getting the fiddly bits of collecting the stats correct. The main downsides are it's a little less discoverable, and only someone with write permissions to the ideas repo can kick it off. It is possible to give individuals that permission, though.

[mdboom]

Closing as DONE. Diffing isn't automated, but is possible with two sets of .json pystats and the latest Tools/scripts/summarize_stats.py script.