bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() #20944

vstinner · 2020-06-17T16:10:42Z

On Linux, skip tests using multiprocessing if the current user cannot
create a file in /dev/shm/ directory. Add the
skip_if_broken_multiprocessing_synchronize() function to the
test.support module.

https://bugs.python.org/issue38377

On Linux, skip tests using multiprocessing if the current user cannot create a file in /dev/shm/ directory. Add the skip_if_broken_multiprocessing_synchronize() function to the test.support module.

vstinner · 2020-06-17T16:14:01Z

Maybe my function is too strict, not all tests require to create a SemLock. But my intent is to avoid coupling tests like test_asyncio with low-level implementation details of multiprocessing.

In practice, only the OpenSUSE build farm should be affected by this change. In general, /dev/shm/ is writable by anyone. Example on Fedora 32:

$ ls -ld /dev/shm/
drwxrwxrwt. 2 root root 40 17 juin  18:13 /dev/shm//

vstinner · 2020-06-17T16:14:29Z

@mcepl: Would it be possible for you to check if my change fix https://bugs.python.org/issue38377 for your use case?

vstinner · 2020-06-17T16:36:59Z

cc @pitrou

mcepl · 2020-06-17T19:30:02Z

On Python 3.8:

[ 1064s] test test_concurrent_futures crashed -- Traceback (most recent call last):
[ 1064s]   File "/home/abuild/rpmbuild/BUILD/Python-3.8.3/Lib/test/libregrtest/runtest.py", line 270, in _runtest_inner
[ 1064s]     refleak = _runtest_inner2(ns, test_name)
[ 1064s]   File "/home/abuild/rpmbuild/BUILD/Python-3.8.3/Lib/test/libregrtest/runtest.py", line 221, in _runtest_inner2
[ 1064s]     the_module = importlib.import_module(abstest)
[ 1064s]   File "/home/abuild/rpmbuild/BUILD/Python-3.8.3/Lib/importlib/__init__.py", line 127, in import_module
[ 1064s]     return _bootstrap._gcd_import(name[level:], package, level)
[ 1064s]   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
[ 1064s]   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
[ 1064s]   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
[ 1064s]   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
[ 1064s]   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
[ 1064s]   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[ 1064s]   File "/home/abuild/rpmbuild/BUILD/Python-3.8.3/Lib/test/test_concurrent_futures.py", line 6, in <module>
[ 1064s]     support.skip_if_broken_multiprocessing_synchronize()
[ 1064s] NameError: name 'support' is not defined

Replacing support. with test.support. made it work.

vstinner · 2020-06-17T21:13:49Z

[ 1064s] NameError: name 'support' is not defined

Well, that's a bug in your backport.

Replacing support. with test.support. made it work.

I understand that this PR fix https://bugs.python.org/issue38377, good!

mcepl · 2020-06-17T21:15:53Z

I understand that this PR fix https://bugs.python.org/issue38377, good!

Yes, it does. Pushing it now to the official openSUSE packages (OBS_dev-shm.patch)

vstinner · 2020-06-17T23:23:40Z

@mcepl: So which approach do you prefer? This PR or your PR #19073?

mcepl · 2020-06-18T10:12:12Z

Obviously this one, it is a way more complete and less hassle to everybody. I’ll close my PR.

vstinner · 2020-06-18T13:02:08Z

Obviously this one, it is a way more complete and less hassle to everybody. I’ll my PR.

Ok, I merged my PR and I'm backporting it to 3.8 and 3.9.

…H-20944) (GH-20962) On Linux, skip tests using multiprocessing if the current user cannot create a file in /dev/shm/ directory. Add the skip_if_broken_multiprocessing_synchronize() function to the test.support module. (cherry picked from commit ddbeb2f)

bedevere-bot · 2020-06-18T15:50:27Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86-64 macOS 3.9 has failed when building commit b1e7361.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/696/builds/119) and take a look at the build logs.
Check if the failure is related to this commit (b1e7361) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/696/builds/119

Failed tests:

test_multiprocessing_forkserver

Failed subtests:

test_resource_tracker_reused - test.test_multiprocessing_forkserver.TestResourceTracker

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

406 tests OK.

10 slowest tests:

test_concurrent_futures: 4 min 25 sec
test_tokenize: 3 min 5 sec
test_unparse: 3 min 1 sec
test_multiprocessing_spawn: 2 min 41 sec
test_multiprocessing_forkserver: 2 min 2 sec
test_lib2to3: 2 min 1 sec
test_unicodedata: 1 min 49 sec
test_asyncio: 1 min 44 sec
test_capi: 1 min 44 sec
test_pickle: 1 min 4 sec

1 test failed:
test_multiprocessing_forkserver

18 tests skipped:
test_devpoll test_epoll test_gdb test_ioctl test_msilib
test_multiprocessing_fork test_ossaudiodev test_smtpnet test_spwd
test_ssl test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

1 re-run test:
test_multiprocessing_forkserver

Total duration: 27 min 13 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/_test_multiprocessing.py", line 5219, in test_resource_tracker_reused
    self.assertTrue(is_resource_tracker_reused)
AssertionError: False is not true

…H-20944) (GH-20962) (GH-20966) On Linux, skip tests using multiprocessing if the current user cannot create a file in /dev/shm/ directory. Add the skip_if_broken_multiprocessing_synchronize() function to the test.support module. (cherry picked from commit ddbeb2f) (cherry picked from commit b1e7361)

bedevere-bot · 2020-06-18T17:22:13Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86-64 macOS 3.8 has failed when building commit e805618.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/273/builds/269) and take a look at the build logs.
Check if the failure is related to this commit (e805618) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/273/builds/269

Failed tests:

test_multiprocessing_forkserver

Failed subtests:

test_resource_tracker_reused - test.test_multiprocessing_forkserver.TestResourceTracker

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

400 tests OK.

10 slowest tests:

test_concurrent_futures: 3 min 13 sec
test_multiprocessing_spawn: 3 min 8 sec
test_tools: 2 min 19 sec
test_tokenize: 2 min 19 sec
test_multiprocessing_forkserver: 2 min
test_lib2to3: 1 min 35 sec
test_asyncio: 1 min 23 sec
test_io: 55.4 sec
test_logging: 50.4 sec
test_pickle: 49.7 sec

1 test failed:
test_multiprocessing_forkserver

22 tests skipped:
test_devpoll test_epoll test_gdb test_idle test_ioctl test_msilib
test_multiprocessing_fork test_ossaudiodev test_smtpnet test_spwd
test_ssl test_startfile test_tcl test_tix test_tk test_ttk_guionly
test_ttk_textonly test_turtle test_winconsoleio test_winreg
test_winsound test_zipfile64

1 re-run test:
test_multiprocessing_forkserver

Total duration: 22 min 6 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.8.billenstein-macos/build/Lib/test/_test_multiprocessing.py", line 5180, in test_resource_tracker_reused
    self.assertTrue(is_resource_tracker_reused)
AssertionError: False is not true

…ythonGH-20944) On Linux, skip tests using multiprocessing if the current user cannot create a file in /dev/shm/ directory. Add the skip_if_broken_multiprocessing_synchronize() function to the test.support module.

bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize()

0edba4a

On Linux, skip tests using multiprocessing if the current user cannot create a file in /dev/shm/ directory. Add the skip_if_broken_multiprocessing_synchronize() function to the test.support module.

vstinner requested review from 1st1, asvetlov and vsajip as code owners June 17, 2020 16:10

the-knights-who-say-ni added the CLA signed label Jun 17, 2020

bedevere-bot added the awaiting core review label Jun 17, 2020

vstinner mentioned this pull request Jun 17, 2020

bpo-38377: multiprocessing.SemLock requires working /dev/shm on Linux. #19073

Closed

vstinner merged commit ddbeb2f into python:master Jun 18, 2020

bedevere-bot removed the awaiting core review label Jun 18, 2020

vstinner deleted the skip_if_broken_multiprocessing_synchronize branch June 18, 2020 12:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() #20944

bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() #20944

vstinner commented Jun 17, 2020 •

edited by bedevere-bot

Loading

vstinner commented Jun 17, 2020

vstinner commented Jun 17, 2020

vstinner commented Jun 17, 2020

mcepl commented Jun 17, 2020 •

edited

Loading

vstinner commented Jun 17, 2020

mcepl commented Jun 17, 2020 •

edited

Loading

vstinner commented Jun 17, 2020

mcepl commented Jun 18, 2020 •

edited

Loading

vstinner commented Jun 18, 2020

bedevere-bot commented Jun 18, 2020

bedevere-bot commented Jun 18, 2020

bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() #20944

bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() #20944

Conversation

vstinner commented Jun 17, 2020 • edited by bedevere-bot Loading

vstinner commented Jun 17, 2020

vstinner commented Jun 17, 2020

vstinner commented Jun 17, 2020

mcepl commented Jun 17, 2020 • edited Loading

vstinner commented Jun 17, 2020

mcepl commented Jun 17, 2020 • edited Loading

vstinner commented Jun 17, 2020

mcepl commented Jun 18, 2020 • edited Loading

vstinner commented Jun 18, 2020

bedevere-bot commented Jun 18, 2020

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Jun 18, 2020

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

vstinner commented Jun 17, 2020 •

edited by bedevere-bot

Loading

mcepl commented Jun 17, 2020 •

edited

Loading

mcepl commented Jun 17, 2020 •

edited

Loading

mcepl commented Jun 18, 2020 •

edited

Loading