-
Notifications
You must be signed in to change notification settings - Fork 220
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PGO (profile-guided optimizations) build for Python #423
Comments
For Windows, you could take a look at the |
+1 |
For reference: https://bugs.python.org/issue24915 |
Are there any implications to compiling Python in this manner? |
The main implication is that building is much longer (mostly due to the training part) :-) |
The package size may be a tiny bit larger too. On my machine it grew from 29 MB to 31.5 MB. I don't think that's important at all. |
I don't think the size increase is a problem. |
So, I have done a little experimenting with the PGO builds. Using your test code above, it achieved a nice 20% (tested with PyBench) overall speedup on my OSX machine. A discussion has been started to possibly release PGO builds in the future for Python 2.7, 3.4, and 3.5 on Linux/OSX. It may be possible to release Windows as well (I hope to figure out PGO on windows in the near future). Thank you for suggesting this. |
My tests with windows PGO builds using Python 3.5 only show a modest overall improvement of ~8%. |
Any recent update on this? :-) |
Also https://bugs.python.org/issue25702 (Link Time Optimizations support for GCC and CLANG) and https://bugs.python.org/issue26359 (CPython build options for out-of-the box performance). |
@tpn do you have any insights on this from the Windows perspective? @pitrou have you or anyone else (e.g. @groutr ) tried this with icc or pgc? Or are these results just for gcc? IIRC there was discussion about gcc, alternatives to gcc, and using a more recent gcc that wouldn't be as widely supported for "legacy hardware" but could get us even better performance for ~80% of users (x86_64). |
Just for gcc. I don't think icc is a reasonable alternative as it can break some expectations in certain cases (see reports at https://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=icc&submit=search&status=-1%2C1%2C2%2C3). I haven't tried with old gcc, all my experiments have been with a reasonably recent gcc (which is required for LTO, by the way). |
Switching to a newer toolchain (see anaconda/docker-images#20) should make it possible to build with a newer gcc on older systems, so, hopefully, the gcc version will no longer be a major limitation. |
Closed, Python 3.6 uses LTO and PGO on all platforms and 3.5 uses LTO and PGO on macOS and Linux. To enable these features on Python 2 I would probably need to back-port lots of changes to the build scripts and I do not want to risk breaking things for people using this legacy Python version. Thanks for the pointers @pitrou. |
Small update to this, LTO and PGO is also enabled now on Python 3.5 on Windows. |
To enable PGO in the Python builds, replace the
make
line in build.sh with the following:On my machine, this makes most benchmarks from the Python benchmarks suite (https://hg.python.org/benchmarks) between 10% and 25% faster.
Note this was done with the Python 3.5.0rc1 build, but it should be quite portable to other Python versions. As for Windows, I'm afraid the way to do it would be entirely different...
The text was updated successfully, but these errors were encountered: