bpo-38061: subprocess uses closefrom() on FreeBSD #19697
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @koobs |
|
PR created from this FreeBSD downstream patch: |
gpshead
approved these changes
Apr 24, 2020
Optimize the subprocess module on FreeBSD using closefrom(). A single close(fd) syscall is cheap, but when sysconf(_SC_OPEN_MAX) is high, the loop calling close(fd) on each file descriptor can take several milliseconds. The workaround on FreeBSD to improve performance was to load and mount the fdescfs kernel module, but this is not enabled by default. Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans (kevans) and Kubilay Kocak (koobs): https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
|
I will check with FreeBSD developers if the authors are correct and check that the fix works as expected. |
koobs
reviewed
Apr 24, 2020
| fdescfs kernel module, but this is not enabled by default. | ||
|
|
||
| Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans (kevans) and | ||
| Kubilay Kocak (koobs): |
There was a problem hiding this comment.
Can remove this, full credit to Ed, Conrad & Kyle
There was a problem hiding this comment.
Even if you didn't write the patch directly, you helped to get it packaged. Thanks @koobs, you deserved to be credited ;-)
|
This change could also add sun to get this supported under Solaris and illumos |
|
I tested this change functionaly using truss: It works as expected: closefrom() is called Without this change, there are way more syscalls (+229 274) :-) ... which is very close to the maximum number of file descriptor: |
Sure. But please propose a separated PR. |
CuriousLearner
added a commit
to CuriousLearner/cpython
that referenced
this pull request
Apr 27, 2020
* 'master' of github.com:python/cpython: (2949 commits) Add files in tests/test_peg_generator to the install target lists (pythonGH-19723) bpo-40398: Fix typing.get_args() for special generic aliases. (pythonGH-19720) bpo-40348: Fix typos in the programming FAQ (pythonGH-19729) bpo-38387: Formally document PyDoc_STRVAR and PyDoc_STR macros (pythonGH-16607) bpo-40401: Remove duplicate pyhash.h include from pythoncore.vcxproj (pythonGH-19725) bpo-40387: Improve queue join() example. (pythonGH-19724) bpo-40396: Support GenericAlias in the typing functions. (pythonGH-19718) Fix typo in Lib/typing.py (pythonGH-19717) Fix typo in object.__format__ docs (pythonGH-19504) bpo-40275: Avoid importing logging in test.support (pythonGH-19601) bpo-40275: Avoid importing socket in test.support (pythonGH-19603) bpo-40275: Avoid importing asyncio in test.support (pythonGH-19600) bpo-40279: Add some error-handling to the module initialisation docs example (pythonGH-19705) closes bpo-40385: Remove Tools/scripts/checkpyc.py (pythonGH-19709) bpo-40334: Add What's New sections for PEP 617 and PEP 585 (pythonGH-19704) bpo-40340: Separate examples more clearly in the programming FAQ (pythonGH-19688) bpo-40360: Deprecate lib2to3 module in light of PEP 617 (pythonGH-19663) bpo-40334: Rewrite test_c_parser to avoid memory leaks (pythonGH-19694) bpo-38061: subprocess uses closefrom() on FreeBSD (pythonGH-19697) bpo-38061: os.closerange() uses closefrom() on FreeBSD (pythonGH-19696) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Optimize the subprocess module on FreeBSD using closefrom().
A single close(fd) syscall is cheap, but when sysconf(_SC_OPEN_MAX)
is high, the loop calling close(fd) on each file descriptor can take
several milliseconds.
The workaround on FreeBSD to improve performance was to load and
mount the fdescfs kernel module, but this is not enabled by default.
Initial patch by Kyle Evans with the help of Kubilay Kocak:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
https://bugs.python.org/issue38061