GH-96421: Insert shim frame on entry to interpreter

[markshannon]

Simplifies RETURN_VALUE, RETURN_GENERATOR and YIELD_VALUE as they no longer need to check if the current frame is the entry frame.

Should allow specialization of FOR_ITER and SEND for generators and coroutines.

Needs docs and news.

Performance impact is about zero

• Issue: Insert shim frames at entries points to the interpreter. #96421

[sweeneyde]

Does this add much extra CPU/stack/memory overhead for calling Python from C? Or alternating py/c/py/c/py calls? Probably not the most important use-cases, but I would hope that they wouldn't pessimize too much.

[markshannon]

Does this add much extra CPU/stack/memory overhead for calling Python from C?

• CPU: A little, but it speeds up the return sequence. The net effect is about zero
• C Stack consumption is increased, but only 80 bytes per call.

[bedevere-bot]

Thanks for making the requested changes!

@brandtbucher: please review the changes made to this pull request.

[brandtbucher]

I don't know if I'll have time to review again today (still need to prepare my talk for the release stream). Is this able to wait until next week? If not, I can try to make time.

[markshannon]

It can wait until next week

[brandtbucher]

Mostly nits on the changes.

Also, I'd like to discuss this vs. something like #98795 again at our stand-up, if that's okay. This still feels pretty heavy/breaky to me compared to that PR (and I'm still not clear what it actually gives us over that approach).

[gvanrossum]

I had some earlier comments pending (not sure if they still apply). But mainly this is a reminder that instructions are now in bytecodes.c, and changes there require running make regen-cases.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @markshannon for commit d113655 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[brandtbucher]

Just a couple of remaining nits.

I'm not sure what kind of review you're looking for... I don't see any technical issues with the implementation, so I'll hit "approve". :)

But I still have reservations about this change being much deeper (and probably more expensive) than what's really necessary right now. I still feel like I haven't really gotten an answer on why this change is more appropriate than something like #98795. So I don't necessarily approve of this approach, but I also won't block you from merging this if you still feel differently.