Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-105699: Add Some Stress Tests for Subinterpreter Creation #106966

Merged
merged 1 commit into from
Jul 22, 2023

Conversation

ericsnowcurrently
Copy link
Member

The tests would have caught the crashes from gh-105699.

interp = interpreters.create()
alive.append(interp)

def test_create_many_threaded(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def test_create_many_threaded(self):
@requires_resource('cpu')
def test_create_many_threaded(self):

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, what is the effect of this decorator? Also, what is it telling readers?

I wouldn't consider this test to be particularly CPU-intensive. We do spin up a couple hundred threads but not so many that this test should take more than a few seconds on a resource-starved host. Likewise, this shouldn't impose a significant memory burden. (Each interpreter uses ~3kB.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cpu - Used for certain CPU-heavy tests.

AFAIK, Conventionally, most CPU-intensive tests consume a few seconds in high-performance computers, we added the flag for low-performance CPUs to skip this test.

I thought that this test look quite CPU-intensive test, so I suggested adding it.

@corona10
Copy link
Member

Left some minor suggestions.

@ambv ambv merged commit adda43d into python:main Jul 22, 2023
19 checks passed
@miss-islington
Copy link
Contributor

Thanks @ericsnowcurrently for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

@bedevere-bot
Copy link

GH-107012 is a backport of this pull request to the 3.12 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 22, 2023
@bedevere-bot bedevere-bot removed the needs backport to 3.12 bug and security fixes label Jul 22, 2023
@ericsnowcurrently ericsnowcurrently deleted the interpreter-stress-tests branch July 24, 2023 15:08
ericsnowcurrently added a commit that referenced this pull request Jul 25, 2023
…H-106966) (gh-107012)

gh-105699: Add some stress tests for subinterpreter creation (GH-106966)
(cherry picked from commit adda43d)

Co-authored-by: Eric Snow <[email protected]>
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.12 has failed when building commit ca42d67.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1159/builds/222) and take a look at the build logs.
  4. Check if the failure is related to this commit (ca42d67) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1159/builds/222

Failed tests:

  • test_interpreters

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 9, done.        
remote: Counting objects:  11% (1/9)        
remote: Counting objects:  22% (2/9)        
remote: Counting objects:  33% (3/9)        
remote: Counting objects:  44% (4/9)        
remote: Counting objects:  55% (5/9)        
remote: Counting objects:  66% (6/9)        
remote: Counting objects:  77% (7/9)        
remote: Counting objects:  88% (8/9)        
remote: Counting objects: 100% (9/9)        
remote: Counting objects: 100% (9/9), done.        
remote: Compressing objects:  20% (1/5)        
remote: Compressing objects:  40% (2/5)        
remote: Compressing objects:  60% (3/5)        
remote: Compressing objects:  80% (4/5)        
remote: Compressing objects: 100% (5/5)        
remote: Compressing objects: 100% (5/5), done.        
remote: Total 5 (delta 4), reused 0 (delta 0), pack-reused 0        
From https://github.com/python/cpython
 * branch            3.12       -> FETCH_HEAD
Note: checking out 'ca42d6720850967698d6e298e37a4225f0c67d8e'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at ca42d67... [3.12] gh-105699: Add some stress tests for subinterpreter creation (GH-106966) (gh-107012)
Switched to and reset branch '3.12'

Could Not Find D:\buildarea\3.12.bolen-windows10\build\Lib\*.pyc
The system cannot find the file specified.
Could Not Find D:\buildarea\3.12.bolen-windows10\build\PCbuild\python*.zip

Debug memory block at address p=000001D24C96FAA0: API '�'
    1731634056723955712 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0x8b *** OUCH
        at p-6: 0x03 *** OUCH
        at p-5: 0x00 *** OUCH
        at p-4: 0xfd
        at p-3: 0xfd
        at p-2: 0xfd
        at p-1: 0xfd
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=180801D24C96FAA0 are Windows fatal exception: access violation

Current thread 0x0000263c (most recent call first):
  File "<frozen importlib._bootstrap_external>", line 1669 in _fill_cache
  File "<frozen importlib._bootstrap_external>", line 1601 in find_spec
  File "<frozen importlib._bootstrap_external>", line 1498 in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1524 in find_spec
  File "<frozen importlib._bootstrap>", line 1168 in _find_spec
  File "<frozen importlib._bootstrap>", line 1228 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1266 in _find_and_load
  File "<frozen importlib._bootstrap>", line 400 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1321 in _handle_fromlist
  File "D:\buildarea\3.12.bolen-windows10\build\Lib\encodings\__init__.py", line 33 in <module>
  File "<frozen importlib._bootstrap>", line 400 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 994 in exec_module
  File "<frozen importlib._bootstrap>", line 841 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1237 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1266 in _find_and_load

Cannot open file 'D:\buildarea\3.12.bolen-windows10\build\test-results.xml' for upload

Could Not Find D:\buildarea\3.12.bolen-windows10\build\PCbuild\python*.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
skip news tests Tests in the Lib/test dir
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants