X Tutup
Skip to content

gh-117648: Improve performance of os.join by replacing map with a direct method …#117654

Merged
erlend-aasland merged 3 commits intopython:mainfrom
nineteendo:speedup-os.path.join-2
Apr 9, 2024
Merged

gh-117648: Improve performance of os.join by replacing map with a direct method …#117654
erlend-aasland merged 3 commits intopython:mainfrom
nineteendo:speedup-os.path.join-2

Conversation

@nineteendo
Copy link
Contributor

@nineteendo nineteendo commented Apr 8, 2024 •
edited
Loading

…call

Benchmark

ntpath.py

script 
::test.bat
@echo off
echo 1 item && python -m timeit -s "import before.ntpath" "before.ntpath.join('foo')" && python -m timeit -s "import after.ntpath" "after.ntpath.join('foo')"
echo 10 items && python -m timeit -s "import before.ntpath; paths = ['foo'] * 10" "before.ntpath.join(*paths)" && python -m timeit -s "import after.ntpath; paths = ['foo'] * 10" "after.ntpath.join(*paths)"
echo 100 items && python -m timeit -s "import before.ntpath; paths = ['foo'] * 100" "before.ntpath.join(*paths)" && python -m timeit -s "import after.ntpath; paths = ['foo'] * 100" "after.ntpath.join(*paths)"
1 item
500000 loops, best of 5: 699 nsec per loop # before
500000 loops, best of 5: 612 nsec per loop # after
# -> 1.14x faster
10 items
50000 loops, best of 5: 5.47 usec per loop # before
50000 loops, best of 5: 5.18 usec per loop # after
# -> 1.06x faster
100 items
5000 loops, best of 5: 54.6 usec per loop # before
5000 loops, best of 5: 51.9 usec per loop # after
# -> 1.05x faster

posixpath.py

script 
# test.sh
echo 1 item && python -m timeit -s "import before.posixpath" "before.posixpath.join('foo')" && python -m timeit -s "import after.posixpath" "after.posixpath.join('foo')"
echo 10 items && python -m timeit -s "import before.posixpath; paths = ['foo'] * 10" "before.posixpath.join(*paths)" && python -m timeit -s "import after.posixpath; paths = ['foo'] * 10" "after.posixpath.join(*paths)"
echo 100 items && python -m timeit -s "import before.posixpath; paths = ['foo'] * 100" "before.posixpath.join(*paths)" && python -m timeit -s "import after.posixpath; paths = ['foo'] * 100" "after.posixpath.join(*paths)"
1 item
1000000 loops, best of 5: 325 nsec per loop # before
1000000 loops, best of 5: 241 nsec per loop # after
# -> 1.35x faster
10 items
100000 loops, best of 5: 3.33 usec per loop # before
100000 loops, best of 5: 3.32 usec per loop # after
# -> no difference
100 items
10000 loops, best of 5: 33.8 usec per loop # before
10000 loops, best of 5: 34.1 usec per loop # after
# -> no difference

@nineteendo @eendebakpt
Improve performance of os.join by replacing map with a direct method …
f53562b 
…call

Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
@bedevere-app bedevere-app bot mentioned this pull request Apr 8, 2024
Closed
@nineteendo nineteendo marked this pull request as ready for review April 9, 2024 05:56
@erlend-aasland erlend-aasland added the performance Performance or resource usage label Apr 9, 2024
erlend-aasland
erlend-aasland approved these changes Apr 9, 2024
View reviewed changes
Copy link
Contributor

@erlend-aasland erlend-aasland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change itself is minimal, code readability/maintainability is not affected, and the speedup seems good enough to warrant an approval.

I still recommend redirecting your enthusiasm towards triaging the existing issues in the bug-tracker and helping fixing real bugs instead of looking for micro-optimisations. See my comment #117634 (comment).

@erlend-aasland erlend-aasland merged commit 99852d9 into python:main Apr 9, 2024
@nineteendo nineteendo deleted the speedup-os.path.join-2 branch April 9, 2024 08:36
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
@nineteendo @eendebakpt @diegorusso
pythongh-117648: Improve performance of os.join (python#117654)
ccbd15b 
Replace map() with a method call in the loop body.

Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@erlend-aasland erlend-aasland erlend-aasland approved these changes

Assignees

No one assigned

Labels

performance Performance or resource usage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nineteendo @erlend-aasland
X Tutup