Byte Pair Encoding

This is a naive implementation of Byte Pair Encoding (BPE) for tokenization of text.

How to use

Clone the repository and run example.py to train and tokenize a sample text.

git clone https://github.com/shahad-mahmud/bpe.git

python example.py

Name		Name	Last commit message	Last commit date
Latest commit History 7 Commits
files		files
.gitignore		.gitignore
README.md		README.md
bpe.py		bpe.py
example.py		example.py
utils.py		utils.py