Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor spm_encode #952

Closed
wants to merge 49 commits into from
Closed

Refactor spm_encode #952

wants to merge 49 commits into from

Conversation

vmarkovtsev
Copy link
Contributor

No description provided.

@vmarkovtsev
Add "nfkc_code" normalization rules
35c2eba
@vmarkovtsev
Add multiple memory and performance optimizations
9e15573 
Improvements measured on 40.6GB chunk of the Pile.
Peak memory consumption: >200GB -> 100GB.
Time to process: 12h -> 2h.
@vmarkovtsev
Add installation instructions and use external abseil
8fc56b5
@vmarkovtsev
Allow switching between internal and external abseil
180a6fb
@vmarkovtsev
Fix the overflow errors
527af0c
@vmarkovtsev
Update the built-in protobuf sources
23c0c09
@vmarkovtsev
Repair the python package build
f64cbf4
@vmarkovtsev
Allow missing tcmalloc
20be455
@vmarkovtsev
Support --verbatim_control_char to mark source code sentences
8d932e5
@vmarkovtsev
Ensure that Encode/Decode restores whitespace in the code
8f26fad
@vmarkovtsev
Fix not adding all-whitespace pairs on --verbatim_control_char
20f12dc
@vmarkovtsev
Get rid of TBB to sort in parallel
6faa2d5 
TBB leak.
@vmarkovtsev
Parallel spm_encode
27f78bd
@vmarkovtsev
Indicate progress in spm_encode
d020431
@vmarkovtsev
Link external protobuf statically
581efe6
@vmarkovtsev
Cache sentence frequencies to a file
4d156ed
@vmarkovtsev
Implement merging cached bpe frequency dicts
e8f8d3b
@vmarkovtsev
Reduce memory pressure in bpe trainer symbols construction
28f7bf1
@vmarkovtsev
Fix memory pressure in sorting final merged bpe cache
4f984b0
@vmarkovtsev
Initialize bpe symbols in parallel
8e28f03
@vmarkovtsev
Successfully load 80% of the Pile
429c830
@vmarkovtsev
Switch to 32-bit bpe pointers
cc6a20f
@vmarkovtsev
Parallelize bpe pairs construction
f637bf0
@vmarkovtsev
Polish UpdateActiveSymbols
aeb2bd7
@vmarkovtsev
Add cache only mode in GetPairSymbol
ebd85f3
@vmarkovtsev
Make bpe position updates parallel
cc8c56d
@vmarkovtsev
Avoid the second overhead of GetCachedPairSymbol()
b6c7666
@vmarkovtsev
Optimize the parallel BPE iterations
17acb6d
@vmarkovtsev
Balance sorting and freq computation
ba48031
@vmarkovtsev
Fix encoding non-verbatim text
150ce9d
joerowell and others added 19 commits October 5, 2023 13:49
@joerowell @vmarkovtsev
Add some install instructions for finding the build path.
d1a4bb4
@kuba-- @rbehjati
Fixing tests (#6)
5cec47d 
* Fixing tests

* Fix src/bpe_model_test.cc (#7)

Co-authored-by: Kuba Podgórski <[email protected]>

---------

Co-authored-by: rbehjati <[email protected]>
Add development documentation
f0821a6
@rbehjati
Merge pull request #10 from poolsideai/devdocs
dc661b8 
Add development docs
@yiyangh-ps
handle 0x01~0x04 delimiters
8985958
@yiyangh-ps
Refactor code
f5f0392
@yiyangh-ps
Process lines fewer than 1000
dde84eb
@yiyangh-ps
Support mixed code-text format
515e65b
@yiyangh-ps
Fix build command
d092158
@yiyangh-ps
Remove unsed eos/bos/verbatim_control_char
a237ae4
@yiyangh-ps
Remove 0x04 from encoding sequence
752a0c0
@yiyangh-ps
Refactor encoding using MixedTextCodeIterator
7f16648
Read from stdin
d16e5da
@yiyangh-ps
Fix a race condition issue
1bce046
Add ReadLineStdin to allow reading from stdin
aa566d3
Use std::shared_ptr and a slow-down mechanism
c4bd5ea
Fixes after review
61265db
Add log lines instead of assert
ce412d7
@vmarkovtsev
Refactor spm_encode_main
2055686
@google-cla Google CLA
Copy link

google-cla bot commented Dec 19, 2023

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@vmarkovtsev
Copy link
Contributor Author

Come on... So sorry.

@vmarkovtsev vmarkovtsev deleted the refactor-encode branch December 19, 2023 15:57
@vmarkovtsev vmarkovtsev restored the refactor-encode branch December 19, 2023 15:57
@vmarkovtsev vmarkovtsev deleted the refactor-encode branch December 19, 2023 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@vmarkovtsev @rbehjati @joerowell @kuba-- @yiyangh-ps