Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimizations for large builds #37

Merged
merged 34 commits into from
Mar 2, 2020
Merged

Optimizations for large builds #37

merged 34 commits into from
Mar 2, 2020

Conversation

aras-p
Copy link
Owner

@aras-p aras-p commented Feb 29, 2020 •
edited
Loading

Redo how the data is gathered & stored to better scale for large builds (ref #10, #30).

TLDR: 3x faster, 15x lower memory usage, 4x smaller file size.

Previously, --stop was just smashing all input JSON files together, and --analyze was parsing that huge JSON file, and performing the analysis. The memory allocator used was a simple "bump a pointer, never free" allocator. On a decent size project (e.g. Unity editor build), that was taking 4.8sec / 5.8GB for stop, and 9.9sec / 7.3GB for analyze (so total 14.7sec, and max 7.3GB memory usage). The resulting file was 1.01GB size.

Now:

  • do the JSON parsing during --stop step into the build events data structures (which includes string/name deduplication), and as a result just store the binary data structure file. --analyze reads that binary file and does no JSON parsing at all now.
  • use regular built-in memory allocator (saves a ton of memory for large analyses).
  • multi-threaded --stop part that parses all the JSON files, using enkiTS task scheduler.
  • replace sajson JSON parser with simdjson.
  • use bytell_hash_map instead of C++ built-in one.
  • use xxHash hash function instead of C++ built-in one.
  • optimize "Template sets" and "Function sets" analysis calculation

On the same Unity editor build: 2.4sec / 0.35GB for stop, and 2.7sec / 0.51GB for analyze (so total 5.1sec, and max 0.5GB memory usage). The resulting file is 280MB size.

However the data files produced by --stop and used by --analyze are no longer compatible with previous versions (previous were just JSON, now custom binary).

@aras-p
Print memory allocated.
01c2e29 
On a snapshot of editor build:
--stop: 4.8s (5834.4MB alloc),
--analyze: 9.9s (7261.7MB alloc).
@aras-p
Optimize: --stop produces binary data dump instead of JSON; JSON pars…
91671fe 
…ing and name deduplication happens during --stop, not during --analyze.

On a snapshot of unity editor build:
--stop: 4.8s 5834MB -> 9.5s 7260MB
--analyze: 9.9s 7262MB -> 4.5s 1270MB
data file: 1.01GB -> 268MB
@aras-p
Encapsulate internal JSON parsing & data building state, saves a bit …
4d4a6e6 
…of memory too:

On a snapshot of unity editor build:
--stop: 9.5s 7260MB -> 9.5s 6968MB
--analyze: 4.5s 1270MB (unchanged)
data file: 268MB (unchanged)
@aras-p
Use precomputed string hashes in nameToIndex map:
b821b02 
On a snapshot of unity editor build:
--stop: 9.5s 6968MB -> 9.3s 6975MB
@aras-p
Replace unordered_map for name deduplication with Malte Skarupke's by…
ff9f42a 
…tell_hash_map (https://probablydance.com/2018/05/28/a-new-fast-hash-table-in-response-to-googles-new-fast-hash-table/)

On a snapshot of unity editor build:
--stop: 9.3s 6975MB -> 8.2s 6984MB
@aras-p
Reuse string buffer memory for reading JSON files into.
151569e 
On a snapshot of unity editor build:
--stop: 8.2s 6984MB -> 7.8s 6071MB
@aras-p
Optimize file writing by using a buffer instead of tiny fwrite calls.
862dd68 
On a snapshot of unity editor build:
--stop: 7.8s 6071MB -> 6.5s 6071MB
@aras-p
Optimize file reading by not doing tiny fread calls.
3009485 
On a snapshot of unity editor build:
--analyze: 4.5s 1270MB -> 3.4s 1519MB
@aras-p
Optimize EmitCollapsedTemplateOpt.
2dfbdd1 
On a snapshot of unity editor build:
--analyze: 3.4s 1519MB -> 3.3s 1437MB
@aras-p
Fixing Linux & Windows builds
e02e42c
ben-craig
ben-craig reviewed Mar 1, 2020
View reviewed changes
Copy link

@ben-craig ben-craig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aras-p doesn't optimize runtime code.
aras-p doesn't optimize build throughput.
aras-p optimizes the build throughput profiler.

I'm now expecting performance patches for whatever perf visualizer you use.

src/Analysis.cpp Show resolved Hide resolved
src/Analysis.cpp Show resolved Hide resolved
aras-p added 18 commits March 1, 2020 11:13
@aras-p
Remove "bump pointer, never free" allocator. It's fast, but huge memo…
bad7fa1 
…ry usage on large codebases is a problem, and is not thread safe either.

On a snapshot of unity editor build:
--stop: 6.5s 6071MB -> 7.0s 493MB
--analyze: 3.3s 1437MB -> 4.7s 779MB
@aras-p
Add enkiTS task scheduler library from master branch 3686b99
1416fb5
@aras-p
Add xxHash hashing library from dev branch 5dd837e
ae993bd
@aras-p
Use arena allocator for nameToIndex string names, and xxHash (XXH64) …
d417f81 
…hash function.

On a snapshot of unity editor build:
--stop: 7.0s 493MB -> 5.4s 466MB; BuildEventsParser::NameToIndex 1.47s -> 960ms
@aras-p
Use arena allocator for all BuildNames and other names produced durin…
055e853 
…g analysis.

On a snapshot of unity editor build:
--stop: 5.4s 466MB -> 5.4s 342MB
--analyze: 4.7s 779MB -> 4.1s 750MB
@aras-p
Replace sajson with simdjson (master branch 140e4dd)
4452f1c
@aras-p
Change code to use simdjson
9659ba6 
On a snapshot of unity editor build:
--stop: 5.4s 342MB -> 5.1s 477MB
@aras-p
Multi-threaded JSON parsing
1a5a87c 
On a snapshot of unity editor build (MacBookPro 2018, 6c/12t):
--stop: 5.1s -> 2.4s
@aras-p
Update Make & CMake projects
70b7430
@aras-p
Set timeout for Github Actions (5min), and test CMake path in Linux b…
00e9ed3 
…uilder too
@aras-p
Fixing cmake test action
c2ed649
@aras-p
Actions: try to use gcc 7 on Linux
d3dc4a2
@aras-p
Actions: try to use gcc 7 on Linux
7ba5db4
@aras-p
Actions: try to use gcc 7 on Linux
6142f41
@aras-p
Windows: switch to VS2019 and fix project files
c009810
@aras-p
Update readme
bbfe83b
@aras-p
Linux: link to -lpthread
7d74e74
@aras-p
Fix double fclose
5ecdb30
@aras-p aras-p mentioned this pull request Mar 2, 2020
Closed
@aras-p aras-p mentioned this pull request Mar 2, 2020
Closed
@aras-p
Make data file have header "magic" and a checksum hash, to avoid blin…
6f4597d 
…dly loading wrong files
@aras-p
Move filename prettifying to --stop phase instead of doing it in --an…
a7418ce 
…alyze

On a snapshot of unity editor build:
--stop: 2.4s -> 2.3s, 368MB
--analyze: 4.4s 750MB -> 4.4s, 673MB
@aras-p
Move name demangling to --stop phase instead of doing it in --analyze
c705536 
On a snapshot of unity editor build:
--stop: 2.3s 368MB -> 2.4s, 348MB
--analyze: 4.4s 673MB -> 3.6s 641MB
@aras-p
Use ska::bytell_hash_map in analysis phase as well.
561c5cb 
On a snapshot of unity editor build:
--stop: unchanged at 2.4s 348MB
--analyze: 3.6s 641MB -> 2.9s 578MB
@aras-p
In "Function sets that took longest to compile / optimize" (EmitColla…
36aa6d5 
…psedTemplateOpt), only actually process templated functions.

This makes it faster (skips over many more functions without attempting to collapse the template name), and also
"fixes" output where function sets are actual sets and not random slow individual functions.

On a snapshot of unity editor build:
--stop: unchanged at 2.4s 348MB
--analyze: 2.9s 578MB -> 2.7s 505MB
@aras-p
Fixing Linux build
258c624
@aras-p aras-p merged commit 5b2550e into master Mar 2, 2020
@aras-p aras-p deleted the optimize-large-builds branch March 3, 2020 06:08
@aras-p aras-p mentioned this pull request Mar 3, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ben-craig ben-craig ben-craig left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@aras-p @ben-craig