Huge memory usage when reading large files

[divaltor]

• Yandex Market Language (YML) for Python version: 0.6.0
• Python version: 3.8.3
• Operating System: Windows 10 2004 (for testing) and Debian 10 (for production)

What I Did

I tried to process a 100 MB file and the processing machine exceeded the memory limit (~2 gb). Is there any way to optimize file reading or processing so that it doesn't go out of memory?

[stefanitsky]

@divaltor hi! Hm, i think you can try this in here:

We can add this to the project if it works, it would be great.

[divaltor]

I will try to implement this, but I think I will have to rewrite most of the library

[divaltor]

@stefanitsky I reworked the processing via iterparse, but it didn't give any result. In the end, the file takes up the same amount of RAM

[stefanitsky]

@stefanitsky I reworked the processing via iterparse, but it didn't give any result. In the end, the file takes up the same amount of RAM

Then i think the problem is that the entire package is built on classes and with such a large file size the memory is overloaded, so you can try to generate the classes through the generator in order to parse into memory gradually as needed, and not all at once.