中文
Libraries & frameworks

gigatoken

marcelroed/gigatoken

A GB/s tokenizer library for language models, compatible with HuggingFace and Tiktoken APIs and designed to accelerate large-scale preprocessing.

Topics