[Windows] Embedded Python Memory Leaks #96853
Comments
|
Thanks in advance |
|
I am actually using Python 3.11 and it is happening. It will be of a tremendous help to see that fixed. |
sweeneyde
removed
the
pending
Can you please provide a full reproducer? Like a whole C file which reproduces the issue? If possible, a regular main() function, not a "WINAPI wWinMain" function. Can you reproduce the issue on other operating systems? How do you link your program? |
|
Thanks a lot for your support.
I am assuming your environment of work is Unix/Linux, I cannot speak about memory leaks in Unix environment since I am not working performing my tests in this unix environment.
The code I sent is what it is, it is that simple:
//This is the main entry point for a C/C++ code copiled for Windows 64bit in Microsoft Visual Studio 2029
//int WINAPI wWinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, PWSTR pCmdLine, int nCmdShow)
int main(...)
{
Py_InitializeEx(0);
Py_FinalizeEx();
Py_InitializeEx(0);
Py_FinalizeEx();
return 0;
}
Put a comment in that line and add the main prototype your compliler/linker likes. Trace that code and you will see how memory leaks.
I repeat, this memory leaks might not happen in Unix/Linux environment, I suggest testing this code in windows enviroment and Python 3.11
Thanks a million,
Alexei
PD: If you have and have permission to share the original code for Py_InitializeEx and Py_FinalizeEx send the code to me and I will help you in troubleshooting this code for Windows
…________________________________
De: Victor Stinner ***@***.***>
Enviado: lunes, 19 de septiembre de 2022 09:51 p. m.
Para: python/cpython ***@***.***>
CC: AlexSoft73 ***@***.***>; Author ***@***.***>
Asunto: Re: [python/cpython] Embedded Python Memory Leaks (Issue #96853)
..More code..
Can you please provide a full reproducer? Like a whole C file which reproduces the issue? If possible, a regular main() function, not a "WINAPI wWinMain" function.
Can you reproduce the issue on other operating systems? How do you link your program?
—
Reply to this email directly, view it on GitHub<#96853 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3C7TGERMLZNFLTC3MUHF2DV7DN6XANCNFSM6AAAAAAQNW7ZHA>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
Thanks for your reply,
Well, i am testing my code with Python 3.11 (64bits), the latest version available and it is leaking memory.
I read the whole #44470 and could not find even the reason for that. if you have the code and have permissio to share it, send me the source code for Py_InitializeEx and Py_FinalizeEx to see If I can help you in troubleshooting the memory leaks.
Thanks in advance,
Alexei
…________________________________
De: Dennis Sweeney ***@***.***>
Enviado: viernes, 16 de septiembre de 2022 09:03 p. m.
Para: python/cpython ***@***.***>
CC: AlexSoft73 ***@***.***>; Author ***@***.***>
Asunto: Re: [python/cpython] Embedded Python Memory Leaks (Issue #96853)
I believe this is fixed in Python 3.11, and was tracked by #44470<#44470>. I doubt there would be a backport, since by my understanding, much of the work to fix this was rather involved.
cc @vstinner<https://github.com/vstinner>
—
Reply to this email directly, view it on GitHub<#96853 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3C7TGDYEICQQOT5OXTTTT3V6TODNANCNFSM6AAAAAAQNW7ZHA>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
It's open source - you don't need permission to see the code for Line 1274 in 4e4bfff |
|
Thanks a lot for the info, I am working on it to see if I can help you guys. |
|
I cannot seem to find all files references in "pylifecycle.c" #include "pycore_bytesobject.h" // _PyBytes_InitTypes() |
|
They are in Include/internal. |
|
Hi Python team, I'd like to compile the python source code in VS2019 to try and debug the initialization process, hence be able to signal what is not release when python is called to finalized. Is there an easy way of getting all files (.h,.c,.cpp.,...) required for this |
|
@AlexSoft73 there's some information about how to download and compile Python at the top of this page https://devguide.python.org/ |
|
Yes, I saw it, I am trying to do it to find the reason for those memory leaks and let you knoqw guys
thanks
…________________________________
De: da-woods ***@***.***>
Enviado: sábado, 24 de septiembre de 2022 06:36 a. m.
Para: python/cpython ***@***.***>
CC: AlexSoft73 ***@***.***>; Mention ***@***.***>
Asunto: Re: [python/cpython] Embedded Python Memory Leaks (Issue #96853)
@AlexSoft73<https://github.com/AlexSoft73> there's some information about how to download and compile Python at the top of this page https://devguide.python.org/
—
Reply to this email directly, view it on GitHub<#96853 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3C7TGAH5PVUCRE7MYIYQQDV72ONDANCNFSM6AAAAAAQNW7ZHA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
For some reason I am having no luck in this:
Git Error:
ssh: connect to host github.com port 22: Connection timed out
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
I am behind a proxy
…________________________________
De: da-woods ***@***.***>
Enviado: sábado, 24 de septiembre de 2022 06:36 a. m.
Para: python/cpython ***@***.***>
CC: AlexSoft73 ***@***.***>; Mention ***@***.***>
Asunto: Re: [python/cpython] Embedded Python Memory Leaks (Issue #96853)
@AlexSoft73<https://github.com/AlexSoft73> there's some information about how to download and compile Python at the top of this page https://devguide.python.org/
—
Reply to this email directly, view it on GitHub<#96853 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3C7TGAH5PVUCRE7MYIYQQDV72ONDANCNFSM6AAAAAAQNW7ZHA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Hi dear python team, when compiling in VS2019 I get the following error: [error] I searched for this file I I cannot locate it, where can I find it please thanks in advance |
|
Good day Python Team, I cannot get the python312.dll built (this is my file of interest to troubleshoot the memory leaks) Thanks in advance, Here I am posting the whole log for python solution (PCbuild.sln) built in VS2019: |
Log for build.bat: |
|
Hi Python team, I manage to get the project compile in MSVS2019. Thanks in advance, |
|
Understanding python when embedded. Please, answer the following question A module is an isolated python execution unit? Meaning each module has its own set of variables? thanks in advance |
Each module has it's own namespace. However variables might be shared between a module. If module B does |
|
Yes, I understand the import part, if you import means you can access what you imported, ie: import moduleA as mA Am I right? |
|
Hi da-woods Do you use python embedded or generate comple applications in Python? Thanks, |
|
I've used embedded Python before but it's been a while since I last set it up. |
|
Hi Python Team Thanks is advance to all those that have helped me with my bigining in this forum. ============= Phase -1 Call 24 to cmalloc (requesting 262168 bytes)================= I know when using python from as an application these memory leaks might have no relevance since the OS will free memory for you upon application is finished. Now when embedding Python we have a different story, especially if we are using Python as child threads of the main application thread. The reason I am asking is, I am no expert in the internal working of Python, I am just pursuing memory leaks and trying to understand why it is happening to try and solve the problem. ============================================== [python] ================= Stack trace ===================
|
|
Hi Python team, In <pylifecycle.c> function: static void finalize_modules(PyThreadState *tstate)
This won't destroy the builtins Thanks in advance |
|
The sample code is as simple as You will see how between this steps memory used is increasing. Usually while using the Python interpreter user don't get to see this since when the Python interpreter application calls Py_FinalizeEx(); and then finish, hence the OS frees whatever memory is unreleased Thanks, |
vstinner
changed the title
|
I also ran Valgrind on the main branch: Valgrind finds a leak of 1 kB: But in fact, this "leak" are just global variables ( |
|
On Windows, memory usage of
It doesn't look like a memory leak to me: the memory usage looks stable. Note: I edited Programs/_testembed.c to have 40 iterations. |
|
@vstinner |
|
Oh, I forgot that I rewrote _testembed_Py_Initialize() to avoid the deprecated Py_Initialize() function :-) The function name is now misleading. |
|
_testembed_Py_Initialize() calls: Py_Initialize() is a thin-wrapper to _PyConfig_InitCompatConfig() + Py_InitializeFromConfig(): |
|
Just in case, I did another test with Py_Initialize(): diff --git a/Programs/_testembed.c b/Programs/_testembed.c
index d635c5a4ab..90550388a3 100644
--- a/Programs/_testembed.c
+++ b/Programs/_testembed.c
@@ -31,7 +31,7 @@ char **main_argv;
/* Use path starting with "./" avoids a search along the PATH */
#define PROGRAM_NAME L"./_testembed"
-#define INIT_LOOPS 4
+#define INIT_LOOPS 40
// Ignore Py_DEPRECATED() compiler warnings: deprecated functions are
// tested on purpose here.
@@ -168,13 +168,18 @@ static int test_repeated_init_exec(void)
fprintf(stderr, "--- Loop #%d ---\n", i);
fflush(stderr);
- _testembed_Py_Initialize();
+ Py_SetProgramName(L"./_testembed");
+ Py_Initialize();
int err = PyRun_SimpleString(code);
Py_Finalize();
if (err) {
return 1;
}
+
+ printf("press a key to continue\n");
+ getchar();
}
+
return 0;
}
For me, that's not a leak. For example, if they would be a constant leak of 0.1 MB per iteration, the memory would increase from 2.7 MB to 6.7 MB. But here it's more or less stable, between 3.1 and 3.5 MB. IMO there is no leak and I suggest to just close the issue. |
erlend-aasland
added
the
pending
|
Hi to all and Thanks for the answers, Sorry I am a bit late I was out of my work bench and while I was able to read thru emails this conversation I had no source files, nor test enviroment with me. The platform where I plan to use python is Windows, I cannot speak for Linux/OS/Unix I am not working currently on those platforms Now, I forked the Python project and recompiled it using msvs2019 and I made a simple modification to 3 functions in The functions I mdified were:
when I finished the application, the counters math should be 0, but the result is 14, this means to me that there were 14 calls to "malloc", "calloc" that never got its counter part "free". MallocCallocCounter: 1123 We can close this issue if this is the common desire. Thanks is n advance |
|
Hi ncoghlan Do you use Windows or Unix based OS? Thanks for your time. |
|
Hi vstinner and thanks for your replay You might not consider a leak going from Loop 1: 2.7 MB to Loop 40: 3.5 MB. I do see a leak there, Stress up your test and take up to 100000 loops to see about the that number. Thanks a lot for your time, |
|
I close the issue. This is no memory leak. It's not perfect, but it's good enough for now :-) Python 3.11 is way better than Python 3.10 and older versions regarding memory leaks when Python is embedded in an application.
You're right: I don't consider it as a leak. Read again what I wrote. If you have a clear reproducer, please open a new issue. A clear reproducer would mean that each Py_Initialize+Py_Finalize would leak a fixed amount of memory LEAK bytes, and that repeat it N times would leak LEAK x N bytes. It's not the case here.
See my comment about pymain_free(): right, Py_Finalize() doesn't clear all global variables. There are a few bytes which remain allocated, but that's very small (1 kB), and this memory is not growing at each iteration. If it's already allocated, Python just uses it at the next iteration. It's not perfect, but it's acceptable for now. If someone can propose a design to release this memory with Py_Initialize+Py_Finalize API, I'm interested. |
|
But, if you close the issue how are you going to see it. I thought this exchange was to try and fix/improve Python code. Thanks any ways, |
|
If someone has a reproducer as I described, please open a new issue. If someone has an idea to fix the small leak of 1 kB, please open a new issue. But the bug described in the issue is not a bug: there is no "leak". |
|
Are you tesdting this is Windows or Unix derived system? Thanks again, Alexei |
My comments where I wrote "On Windows", "Task Manager", "_testembed.exe", etc. are on Windows. |
|
Ok vstinner if you think python has no leaks specially, the memory allocated for globals which makes the Python dll not thread safe, and still don't believe the email where I detected that there are 14 calls requesting memory which memory is never released, that tells me, you use Python as a standalone application, hence never going to see this issue. Any ways, thanks |
|
I'm still investigating this, as 6415e2e is a 3.11 specific change that dropped the embedding tests' coverage of While I could just add that apparently missing call (along with a new embedding test to specifically cover the convenience/compatibility APIs) and call it done, I'm first seeing if I can find a diagnostic that can clearly track whether the strings in the config struct (which are allocated with The refcount tracking and allocated block tracking don't pick those up since they only cover pymalloc and actual Python objects, so I'll be checking valgrind next. If that still doesn't clearly demonstrate a difference between |
|
OK, the valgrind results are identical regardless of whether There are a handful of unmatched What you do get over repeated invocations is increased memory fragmentation in the overall process, which can result in the kind of memory creep reported here without being a true leak. |
* As most of `test_embed` now uses `Py_InitializeFromConfig`, add a specific test case to cover `Py_Initialize(Ex)` * Rename `_testembed` init helper to clarify the API used * Add a `PyConfig_Clear` call in `Py_InitializeEx` to make the code more obviously correct (it already didn't leak as none of the dynamically allocated config fields were being populated, but it's clearer if the wrappers follow the documented API usage guidelines)
I tried once to account PyMem_RawMalloc() in "allocated blocks": https://bugs.python.org/issue26850 But with this change, Python said that it leaked memory. I gave up in 2016. In the meanwhile, many "leaks" were fixed: Python now releases most memory at exit. |
|
Thanks ncoghlan for your replay and interest in getting Python improved, I believe it all lies in the Python startup global configuration data not being released in Py_Finalize(); See if this rings you a bell, I've been tracing the allocations. The following stack frame is one where I see (262168 ==> 256Kb) allocated but never released, happening in Stack frame:
|
|
Hi @ncoghlan and thanks again About this comment, There are a handful of unmatched PyMem_RawMalloc calls (related to _Py_path_config as @vstinner noted above), but those don't get repeated (later interpreter instances reuse the previous allocations) so the total memory use doesn't grow over time. I consider the above partially true, partially false, the thing lies here (This is a subject I wanted to talk about after we got resolved the memory issues to go one step at e time). There is a difference when we use python3.dll or python3XX.dll, The one most important to me is: With python3.dll is posible late binding, hence I embed python functionality use it and when done with python release the pythin3.dll and I am finish with python till I need it again. It is here where I'd like to see python releasing all the resources it took from the OS. Why? well next time the same application (still running) needs to use Python, then load Python3.dll again, all those global variables left by previous load become memory leaks. Please, if you don't understand something of the above, let me know, it is a bit tricky ti explain to me. |
* As most of `test_embed` now uses `Py_InitializeFromConfig`, add a specific test case to cover `Py_Initialize` (and `Py_InitializeEx`) * Rename `_testembed` init helper to clarify the API used * Add a `PyConfig_Clear` call in `Py_InitializeEx` to make the code more obviously correct (it already didn't leak as none of the dynamically allocated config fields were being populated, but it's clearer if the wrappers follow the documented API usage guidelines)
…-98212) * As most of `test_embed` now uses `Py_InitializeFromConfig`, add a specific test case to cover `Py_Initialize` (and `Py_InitializeEx`) * Rename `_testembed` init helper to clarify the API used * Add a `PyConfig_Clear` call in `Py_InitializeEx` to make the code more obviously correct (it already didn't leak as none of the dynamically allocated config fields were being populated, but it's clearer if the wrappers follow the documented API usage guidelines) (cherry picked from commit 05e4886) Co-authored-by: Nick Coghlan <[email protected]>
* As most of `test_embed` now uses `Py_InitializeFromConfig`, add a specific test case to cover `Py_Initialize` (and `Py_InitializeEx`) * Rename `_testembed` init helper to clarify the API used * Add a `PyConfig_Clear` call in `Py_InitializeEx` to make the code more obviously correct (it already didn't leak as none of the dynamically allocated config fields were being populated, but it's clearer if the wrappers follow the documented API usage guidelines) (cherry picked from commit 05e4886) Co-authored-by: Nick Coghlan <[email protected]>
erlend-aasland
removed
the
pending
Hi dear Python team,
I am a building a c/c++ application for general purpose and I would like to have in the application the possibility of running “Python” scripting code. Python has proved to be a good programming language to deal with heavy Math as in AI.
Now when I embed Python in my application things go perfect, except for memory leaks!
As a sample by just issuing the following code leaves memory leaks of around (2Mb):
Is there something I must do to cause Python to release all resources allocated
The text was updated successfully, but these errors were encountered: