Skip to content

A byte-level decoder architecture that matches the performance of tokenized Transformers.

Notifications You must be signed in to change notification settings

kjslag/spacebyte

Repository files navigation

SpaceByte

This is the implementation of SpaceByte used in SpaceByte: Towards Deleting Tokenization from Large Language Modeling. SpaceByte is a tokenization-free large language model (LLM) that uses multiscale modeling at the byte and word levels to model language with the same performance as standard LLM architectures that instead use tokenization. See the reproduce directory for instructions for reproducing the results in our paper.

SpaceByte architecture schematic

About

A byte-level decoder architecture that matches the performance of tokenized Transformers.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published