bpo-36867: Make semaphore_tracker track other system resources.

[pierreglaser]

Primary Purpose

This PR proposes to extend the semaphore_tracker range of action to other system resources.
Currently in python3.8, if a process that created a shared_memory object gets violently terminated (with kill for example), the memory segment will not be properly released until the next machine reboot. This will be problematic for long-running clusters used by many different users.

The primary reason of this PR is thus to extend the semaphore_tracker to track also shared_memory segments. The semaphore_tracker module gets consequently renamed resource_tracker.

Extension to a potentially public API

Additionally, the design of the resource_tracker is more generic, and possibly suggests a new public API to enable the tracking other named-resources. For now, the private _CLEANUP_FUNCS dict references all the types that can possibly be tracked.

To start tracking a new type, the resource_tracker only needs to know which cleanup routine to apply to each leaked instance of this type after the python session ends. The public API would thus consist of a function like resource_tracker.register_resource_type(resource_type, cleanup_func), where:

• resource_type is a string
• cleanup_func must be a callable taking a single string as an argument.

Under the hood, register_resource_type would populate the CLEANUP_FUNCS with the new (type, cleanup_func) record.

Operating System Availability

To ease the review process of this PR. the resource_tracker is now POSIX only. I however have a functional windows implementation that I will provide upon demand :)

pinging @pitrou @ogrisel @tomMoral

@pablogsal this may interest you as well :)

https://bugs.python.org/issue36867

[pitrou]

You may want to take a look at the CI failures.

[pierreglaser]

Yes: turns out we don't want a Finalizer for SharedMemory ATM (as opposed to SemLock)

[pitrou]

This PR looks mostly good to me. I have just one comment and a question.

[pierreglaser]

Some more comment about OS availability: On windows, it looks that even with no resource_tracker, abruptly terminating process does not leak shared_memory. So as long as the resource_tracker only tracks SemLocks and shared_memory objects, it is useless for windows.
The tests added in pierreglaser#3 make sure no shared memory leakage happens on all platform, independently of the resource_tracker.

[pitrou]

Indeed, it seems that shared memory on Windows is more cleverly implemented than on Unix.

[pitrou]

cc @applio just in case he wants to confirm.

[pitrou]

A couple more nits :-)

[bedevere-bot]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be poked with soft cushions!

[pierreglaser]

I have made the requested changes; please review again.
(just making the bot happy)

[bedevere-bot]

Thanks for making the requested changes!

@pitrou: please review the changes made to this pull request.

[pierreglaser]

Also @pitrou, before potentially merging, any opinion on pierreglaser#3? It adds shared-memory leaking tests for all platforms, independently of whether the resource_tracker exist for this platform or not.

[pierreglaser]

Note also that there is still work to do to properly release shared memory segments created with a SharedMemoryManager after an abrupt interruption of a python session, but this is independent of this PR (but not independent of #12483)

[pitrou]

Thanks @pierreglaser !

[pierreglaser]

Thanks @pitrou for the infallible reviews ;)