Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Encode file remainder #951

Closed
wants to merge 35 commits into from
Closed

Encode file remainder #951

wants to merge 35 commits into from

Conversation

vmarkovtsev
Copy link
Contributor

No description provided.

@vmarkovtsev
Add "nfkc_code" normalization rules
35c2eba
@vmarkovtsev
Add multiple memory and performance optimizations
9e15573 
Improvements measured on 40.6GB chunk of the Pile.
Peak memory consumption: >200GB -> 100GB.
Time to process: 12h -> 2h.
@vmarkovtsev
Add installation instructions and use external abseil
8fc56b5
@vmarkovtsev
Allow switching between internal and external abseil
180a6fb
@vmarkovtsev
Fix the overflow errors
527af0c
@vmarkovtsev
Update the built-in protobuf sources
23c0c09
@vmarkovtsev
Repair the python package build
f64cbf4
@vmarkovtsev
Allow missing tcmalloc
20be455
@vmarkovtsev
Support --verbatim_control_char to mark source code sentences
8d932e5
@vmarkovtsev
Ensure that Encode/Decode restores whitespace in the code
8f26fad
@vmarkovtsev
Fix not adding all-whitespace pairs on --verbatim_control_char
20f12dc
@vmarkovtsev
Get rid of TBB to sort in parallel
6faa2d5 
TBB leak.
@vmarkovtsev
Parallel spm_encode
27f78bd
@vmarkovtsev
Indicate progress in spm_encode
d020431
@vmarkovtsev
Link external protobuf statically
581efe6
@vmarkovtsev
Cache sentence frequencies to a file
4d156ed
@vmarkovtsev
Implement merging cached bpe frequency dicts
e8f8d3b
@vmarkovtsev
Reduce memory pressure in bpe trainer symbols construction
28f7bf1
@vmarkovtsev
Fix memory pressure in sorting final merged bpe cache
4f984b0
@vmarkovtsev
Initialize bpe symbols in parallel
8e28f03
@vmarkovtsev
Successfully load 80% of the Pile
429c830
@vmarkovtsev
Switch to 32-bit bpe pointers
cc6a20f
@vmarkovtsev
Parallelize bpe pairs construction
f637bf0
@vmarkovtsev
Polish UpdateActiveSymbols
aeb2bd7
@vmarkovtsev
Add cache only mode in GetPairSymbol
ebd85f3
@vmarkovtsev
Make bpe position updates parallel
cc8c56d
@vmarkovtsev
Avoid the second overhead of GetCachedPairSymbol()
b6c7666
@vmarkovtsev
Optimize the parallel BPE iterations
17acb6d
@vmarkovtsev
Balance sorting and freq computation
ba48031
@vmarkovtsev
Fix encoding non-verbatim text
150ce9d
@google-cla Google CLA
Copy link

google-cla bot commented Dec 19, 2023

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@vmarkovtsev vmarkovtsev mentioned this pull request Dec 19, 2023
Closed
@vmarkovtsev vmarkovtsev deleted the encode-file-remainder branch December 19, 2023 14:52
@vmarkovtsev
Copy link
Contributor Author

Sorry for the mess, I misclicked on the fork

@vmarkovtsev vmarkovtsev restored the encode-file-remainder branch December 19, 2023 14:53
@vmarkovtsev vmarkovtsev deleted the encode-file-remainder branch December 19, 2023 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@vmarkovtsev @rbehjati @joerowell @kuba--