Don't run mypy self-test?

[srittau]

I suggest to turn off the mypy self test in CI. These tests are the slowest tests by far, taking over 20 minutes to complete, compared to the 7 minutes it takes the pytest tests to complete. When they fail, they also require the feared "typeshed/mypy" shuffle that only mypy/typeshed maintainers (effectively Jelle) can do.

On the other hand, I can't remember the last time these tests flagged a genuine problem with the stubs. (But I don't have the best memory.) If they fail, it's usually due to a tight coupling between the self test and the way things are typed in typeshed.

As with #4332, the main disadvantage of not running those tests is a higher change of failure of the typeshed sync for mypy failing. But I'd argue that it is easier to resolve at sync time than to do the whole shuffle.

[srittau]

@JelleZijlstra @ilevkivskyi @JukkaL

[JukkaL]

What about only running mypy self check, as discussed in #4332? It runs pretty quickly.

I assume we'd still verify that mypy can type check the stubs without errors?

I wonder how fast we could make the CI. For example, it would be nice if the CI would run in, say, under 5 minutes for most PRs. Also, once we get to a certain level of CI performance, we can have a policy of rejecting new tests that take longer than that to run, as otherwise things tend to regress over time.

[srittau]

Running mypy over the stubs would not be affected. I like the idea of running the self check, but not the test suite. I especially like Jelle's suggestion for a black-prime-like system. Testing stubs against real work code bases is very useful.

[srittau]

To clarify my suggestion:

• Check stubs with mypy against various Python versions and platforms (currently the various "mypy" jobs on CI): Keep (possibly pinning the mypy version).
• Check the mypy code base (currently one part of the "mypy self test" job on CI): Keep, possibly extend to other code bases (possibly using a released mypy version).
• Run mypy's test suite (currently the other part of the "mypy self test" job on CI): Remove.

[JelleZijlstra]

To make CI faster, perhaps we can somehow set up a system that checks only the files changed by a PR; that could especially make sense once we move to modular typeshed.

[srittau]

#4337 is a draft PR that adds a "real" mypy self test. For now it just renames (but still runs) the existing self test to "mypy test suite". Interestingly, it flags a few things in mypy's code base.

[hauntsaninja]

I got #4337 green and merged. I propose for now we change "mypy test suite" to run a subset of tests that are more relevant to typeshed: python/mypy#9638
This should speed things up enough that slowness is hopefully no longer a reason to nix the test completely. I agree that the shuffle is annoying when it's needed, but the typeshed/mypy maintainer bus factor is now two.
Finally, we do check mypy's code base as part of mypy primer, so perhaps once we've figured out what the best way to integrate it into CI is (#4672) we could consider dropping "mypy checks mypy" as redundant (depending on how strict / visible mypy_primer failures are).