bpo-41993: Fix possible issues in remove_module() #22631

serhiy-storchaka · 2020-10-10T10:57:47Z

https://bugs.python.org/issue41993

Python/import.c

vstinner

LGTM.

miss-islington · 2020-10-11T13:51:11Z

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8, 3.9.
🐍🍒⛏🤖

bedevere-bot · 2020-10-11T13:51:26Z

GH-22647 is a backport of this pull request to the 3.9 branch.

miss-islington · 2020-10-11T13:51:27Z

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.8 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 8287aadb75f6bd0154996424819334cd3839707c 3.8

* PyMapping_HasKey() is not safe because it silences all exceptions and can return incorrect result. * Informative exceptions from PyMapping_DelItem() are overridden with RuntimeError and the original exception raised before calling remove_module() is lost. * There is a race condition between PyMapping_HasKey() and PyMapping_DelItem(). (cherry picked from commit 8287aad) Co-authored-by: Serhiy Storchaka <[email protected]>

miss-islington · 2020-10-11T14:05:08Z

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

miss-islington · 2020-10-11T14:05:09Z

Sorry @serhiy-storchaka, I had trouble checking out the 3.8 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 8287aadb75f6bd0154996424819334cd3839707c 3.8

* PyMapping_HasKey() is not safe because it silences all exceptions and can return incorrect result. * Informative exceptions from PyMapping_DelItem() are overridden with RuntimeError and the original exception raised before calling remove_module() is lost. * There is a race condition between PyMapping_HasKey() and PyMapping_DelItem().. (cherry picked from commit 8287aad) Co-authored-by: Serhiy Storchaka <[email protected]>

bedevere-bot · 2020-10-11T15:53:08Z

GH-22649 is a backport of this pull request to the 3.8 branch.

* PyMapping_HasKey() is not safe because it silences all exceptions and can return incorrect result. * Informative exceptions from PyMapping_DelItem() are overridden with RuntimeError and the original exception raised before calling remove_module() is lost. * There is a race condition between PyMapping_HasKey() and PyMapping_DelItem(). (cherry picked from commit 8287aad) Co-authored-by: Serhiy Storchaka <[email protected]>

bedevere-bot · 2020-10-14T09:23:24Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 3.9 has failed when building commit 391a544.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/490/builds/77) and take a look at the build logs.
Check if the failure is related to this commit (391a544) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/490/builds/77

Failed tests:

test_multiprocessing_spawn

Failed subtests:

test_put - test.test_multiprocessing_spawn.WithProcessesTestQueue

Summary of the results of the build (if available):

== Tests result: FAILURE then ENV CHANGED ==

410 tests OK.

10 slowest tests:

test_concurrent_futures: 3 min
test_capi: 2 min 34 sec
test_multiprocessing_spawn: 2 min 22 sec
test_peg_generator: 2 min 10 sec
test_unparse: 1 min 37 sec
test_tokenize: 1 min 36 sec
test_gdb: 1 min 32 sec
test_subprocess: 1 min 22 sec
test_multiprocessing_forkserver: 1 min 17 sec
test_lib2to3: 1 min 17 sec

1 test altered the execution environment:
test_asyncio

14 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_nis
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

1 re-run test:
test_multiprocessing_spawn

Total duration: 10 min 55 sec

Click to see traceback logs

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/multiprocessing/synchronize.py", line 110, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/test/_test_multiprocessing.py", line 938, in test_put
    self.assertEqual(queue_empty(queue), False)
AssertionError: True != False


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 321, in __del__
    self.close()
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 316, in close
    self._ssl_protocol._start_shutdown()
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 590, in _start_shutdown
    self._abort()
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 731, in _abort
    self._transport.abort()
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/asyncio/selector_events.py", line 680, in abort
    self._force_close(None)
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/asyncio/selector_events.py", line 731, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/asyncio/base_events.py", line 746, in call_soon
    self._check_closed()
  File "/home/dje/cpython-buildarea/3.9.edelsohn-rhel8-z/build/Lib/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

* PyMapping_HasKey() is not safe because it silences all exceptions and can return incorrect result. * Informative exceptions from PyMapping_DelItem() are overridden with RuntimeError and the original exception raised before calling remove_module() is lost. * There is a race condition between PyMapping_HasKey() and PyMapping_DelItem().

bpo-41993: Fix possible issues in remove_module()

d017408

serhiy-storchaka added type-bug An unexpected behavior, bug, or error needs backport to 3.8 only security fixes needs backport to 3.9 only security fixes labels Oct 10, 2020

serhiy-storchaka requested a review from vstinner October 10, 2020 10:57

serhiy-storchaka requested review from brettcannon, encukou, ericsnowcurrently, ncoghlan and warsaw as code owners October 10, 2020 10:57

the-knights-who-say-ni added the CLA signed label Oct 10, 2020

bedevere-bot added the awaiting core review label Oct 10, 2020

vstinner reviewed Oct 11, 2020

View reviewed changes

Python/import.c Show resolved Hide resolved

serhiy-storchaka added 2 commits October 11, 2020 15:33

Merge branch 'master' into import-remove_module

a6f86e7

Update comment.

39b3e68

vstinner approved these changes Oct 11, 2020

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Oct 11, 2020

serhiy-storchaka merged commit 8287aad into python:master Oct 11, 2020

bedevere-bot removed the awaiting merge label Oct 11, 2020

serhiy-storchaka deleted the import-remove_module branch October 11, 2020 13:51

bedevere-bot removed the needs backport to 3.9 only security fixes label Oct 11, 2020

miss-islington assigned serhiy-storchaka Oct 11, 2020

serhiy-storchaka added needs backport to 3.8 only security fixes and removed needs backport to 3.8 only security fixes labels Oct 11, 2020

bedevere-bot removed the needs backport to 3.8 only security fixes label Oct 11, 2020

serhiy-storchaka removed their assignment Dec 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-41993: Fix possible issues in remove_module() #22631

bpo-41993: Fix possible issues in remove_module() #22631

serhiy-storchaka commented Oct 10, 2020 •

edited by bedevere-bot

Loading

vstinner left a comment

miss-islington commented Oct 11, 2020

bedevere-bot commented Oct 11, 2020

miss-islington commented Oct 11, 2020

miss-islington commented Oct 11, 2020

miss-islington commented Oct 11, 2020

bedevere-bot commented Oct 11, 2020

bedevere-bot commented Oct 14, 2020

bpo-41993: Fix possible issues in remove_module() #22631

bpo-41993: Fix possible issues in remove_module() #22631

Conversation

serhiy-storchaka commented Oct 10, 2020 • edited by bedevere-bot Loading

vstinner left a comment

Choose a reason for hiding this comment

miss-islington commented Oct 11, 2020

bedevere-bot commented Oct 11, 2020

miss-islington commented Oct 11, 2020

miss-islington commented Oct 11, 2020

miss-islington commented Oct 11, 2020

bedevere-bot commented Oct 11, 2020

bedevere-bot commented Oct 14, 2020

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

serhiy-storchaka commented Oct 10, 2020 •

edited by bedevere-bot

Loading