PGO (profile-guided optimizations) build for Python

[pitrou]

To enable PGO in the Python builds, replace the make line in build.sh with the following:

# The following is basically "make profile-opt", but with additional
# flexibility if you want to add e.g. "-j4" when building

# Tests to exclude from the profiling run (too slow, or crashy)
EXCLUDE_TESTS="test_gdb test_subprocess test_faulthandler \
    test_multiprocessing test_multiprocessing_fork \
    test_multiprocessing_forkserver test_multiprocessing_spawn \
    test_multiprocessing_main_handling test_signal \
    test_concurrent_futures"

PROFILE_TASK="-m test.regrtest -w -uall,-audio -x $EXCLUDE_TESTS"

# Build with support for profile generation
make build_all_generate_profile
# Run workload to generate profile data
# Errors in the test suite are ignored
make run_profile_task PROFILE_TASK="$PROFILE_TASK" || true
# Rebuild with profile guided optimizations
make clean
make build_all_use_profile

# End "make profile-opt"

On my machine, this makes most benchmarks from the Python benchmarks suite (https://hg.python.org/benchmarks) between 10% and 25% faster.

Note this was done with the Python 3.5.0rc1 build, but it should be quite portable to other Python versions. As for Windows, I'm afraid the way to do it would be entirely different...

[pitrou]

For Windows, you could take a look at the PCbuild/build_pgo.bat script in the CPython source tree. You'll have to modify the %job% variable to contain the right training workload (the PROFILE_TASK in the snippet above).

[ahmadia]

+1

[groutr]

For reference: https://bugs.python.org/issue24915

[groutr]

Are there any implications to compiling Python in this manner?

[pitrou]

The main implication is that building is much longer (mostly due to the training part) :-)

[pitrou]

The package size may be a tiny bit larger too. On my machine it grew from 29 MB to 31.5 MB. I don't think that's important at all.

[groutr]

I don't think the size increase is a problem.
I can test building our python with this. I'll report back when it's finished.

[groutr]

So, I have done a little experimenting with the PGO builds. Using your test code above, it achieved a nice 20% (tested with PyBench) overall speedup on my OSX machine. A discussion has been started to possibly release PGO builds in the future for Python 2.7, 3.4, and 3.5 on Linux/OSX. It may be possible to release Windows as well (I hope to figure out PGO on windows in the near future).

Thank you for suggesting this.

[groutr]

My tests with windows PGO builds using Python 3.5 only show a modest overall improvement of ~8%.
EDIT: I forgot to exclude the tests that are excluded in the first post. The new builds show ~15% improvement.

[pitrou]

Any recent update on this? :-)

[pitrou]

Also https://bugs.python.org/issue25702 (Link Time Optimizations support for GCC and CLANG) and https://bugs.python.org/issue26359 (CPython build options for out-of-the box performance).

[ijstokes]

@tpn do you have any insights on this from the Windows perspective?

@pitrou have you or anyone else (e.g. @groutr ) tried this with icc or pgc? Or are these results just for gcc? IIRC there was discussion about gcc, alternatives to gcc, and using a more recent gcc that wouldn't be as widely supported for "legacy hardware" but could get us even better performance for ~80% of users (x86_64).

[pitrou]

Just for gcc. I don't think icc is a reasonable alternative as it can break some expectations in certain cases (see reports at https://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=icc&submit=search&status=-1%2C1%2C2%2C3).

I haven't tried with old gcc, all my experiments have been with a reasonably recent gcc (which is required for LTO, by the way).

[insertinterestingnamehere]

Switching to a newer toolchain (see anaconda/docker-images#20) should make it possible to build with a newer gcc on older systems, so, hopefully, the gcc version will no longer be a major limitation.

[mingwandroid]

Closed, Python 3.6 uses LTO and PGO on all platforms and 3.5 uses LTO and PGO on macOS and Linux.

To enable these features on Python 2 I would probably need to back-port lots of changes to the build scripts and I do not want to risk breaking things for people using this legacy Python version. Thanks for the pointers @pitrou.

[mingwandroid]

Small update to this, LTO and PGO is also enabled now on Python 3.5 on Windows.