bpe

Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

#自然语言处理#Unsupervised text tokenizer focused on computational efficiency

#计算机科学#The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (o1, o3, o4, gpt-4o, gpt-4, etc.). Port of OpenAI's tiktoken with additional features.

Ready-made tokenizer library for working with GPT and tiktoken

#自然语言处理#Fast and customizable text tokenization library with BPE and SentencePiece support

#自然语言处理#Explains nlp building blocks in a simple manner.

#自然语言处理#Byte Pair Encoding for Python!

nfelib - bindings Python para e ler e gerir XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e

Fast bare-bones BPE for modern tokenizer training

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

#计算机科学#Machine Learning for Phishing Website Detection

Subword Encoding in Lattice LSTM for Chinese Word Segmentation

Simple-to-use scoring function for arbitrarily tokenized texts.

#大语言模型#GPT3 encoder & decoder tool written in Swift

Kotlin multiplatform BPE tokenizer library for OpenAI models

BBPE 底层实现

#自然语言处理#Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

High performance unsupervised text tokenization for Ruby

✂️ OpenAI's tiktoken tokenizer written in Go

Learning BPE embeddings by first learning a segmentation model and then training word2vec