gh-117151: increase default buffer size of shutil.copyfileobj() to 25…

…6k. (GH-119783) * gh-117151: increase default buffer size of shutil.copyfileobj() to 256k. it was set to 16k in the 1990s. it was raised to 64k in 2019. the discussion at the time mentioned another 5% improvement by raising to 128k and settled for a very conservative setting. it's 2024 now, I think it should be revisited to match modern hardware. I am measuring 0-15% performance improvement when raising to 256k on various types of disk. there is no downside as far as I can tell. this function is only intended for sequential copy of full files (or file like objects). it's the typical use case that benefits from larger operations. for reference, I came across this function while trying to profile pip that is using it to copy files when installing python packages. * add news --------- Co-authored-by: rmorotti <[email protected]>
python · Oct 4, 2024 · 6efd95c · 6efd95c
1 parent 8bcf118
commit 6efd95c
Show file tree

Hide file tree

Showing 2 changed files with 4 additions and 1 deletion.
diff --git a/Lib/shutil.py b/Lib/shutil.py
@@ -44,7 +44,7 @@
 else:
     _winapi = None
 
-COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 64 * 1024
+COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 256 * 1024
 # This should never be removed, see rationale in:
 # https://bugs.python.org/issue43743#msg393429
 _USE_CP_SENDFILE = (hasattr(os, "sendfile")

diff --git a/Misc/NEWS.d/next/Library/2024-10-03-05-00-25.gh-issue-117151.Prdw_W.rst b/Misc/NEWS.d/next/Library/2024-10-03-05-00-25.gh-issue-117151.Prdw_W.rst
@@ -0,0 +1,3 @@
+The default buffer size used by :func:`shutil.copyfileobj` has been
+increased from 64k to 256k on non-Windows platforms. It was already larger
+on Windows.