{"id":75,"library":"tiktoken","title":"tiktoken","description":"Fast BPE tokenizer from OpenAI, written in Rust. Used to count tokens and encode/decode text for OpenAI models. 3-6x faster than comparable Python tokenizers. Does NOT call any API — purely local computation. Requires a Rust compiler at build time on platforms without pre-built wheels. Package name and import name are both 'tiktoken'.","status":"active","version":"0.12.0","language":"python","source_language":"en","source_url":"https://github.com/openai/tiktoken","tags":["tiktoken","openai","tokenizer","bpe","token-counting","gpt-4o","cl100k","o200k","context-window"],"install":[{"cmd":"pip install tiktoken","lang":"bash","label":"Standard install"},{"cmd":"pip install tiktoken --no-binary tiktoken","lang":"bash","label":"Build from source (requires Rust toolchain)"}],"dependencies":[{"reason":"Required. Used for the BPE pre-tokenization patterns.","package":"regex","optional":false},{"reason":"Required. Used to download encoding vocab files on first use (fetches from OpenAI CDN). Can be disabled for offline use via TIKTOKEN_CACHE_DIR.","package":"requests","optional":false},{"reason":"Optional. Used as an alternative backend for fetching encoding files.","package":"blobfile","optional":true}],"imports":[{"note":"get_encoding() takes an encoding name (e.g. 'o200k_base'), NOT a model name. Passing a model name raises KeyError.","wrong":"import tiktoken; enc = tiktoken.get_encoding('gpt-4o')","symbol":"get_encoding","correct":"import tiktoken; enc = tiktoken.get_encoding('o200k_base')"},{"note":"encoding_for_model() takes a model name and returns the correct encoding. Preferred over hardcoding encoding names.","symbol":"encoding_for_model","correct":"import tiktoken; enc = tiktoken.encoding_for_model('gpt-4o')"}],"quickstart":{"code":"import tiktoken\n\n# Get encoding by model name (recommended)\nenc = tiktoken.encoding_for_model('gpt-4o')  # returns o200k_base\n\n# Or get encoding directly by name\nenc = tiktoken.get_encoding('o200k_base')\n\n# Encode text → list of token integers\ntokens = enc.encode('Hello, world!')\nprint(tokens)        # [9906, 11, 1917, 0]\nprint(len(tokens))   # 4\n\n# Decode tokens → string\ntext = enc.decode(tokens)\nprint(text)  # 'Hello, world!'\n\n# Count tokens for a chat message (accounts for message overhead)\ndef count_chat_tokens(messages: list[dict], model: str = 'gpt-4o') -> int:\n    enc = tiktoken.encoding_for_model(model)\n    tokens_per_message = 3  # every message has <|im_start|>, role, <|im_end|>\n    tokens_per_name = 1\n    total = 0\n    for msg in messages:\n        total += tokens_per_message\n        for key, value in msg.items():\n            total += len(enc.encode(value))\n            if key == 'name':\n                total += tokens_per_name\n    total += 3  # reply is primed with <|im_start|>assistant\n    return total\n\nmessages = [{'role': 'user', 'content': 'How many tokens is this?'}]\nprint(count_chat_tokens(messages))  # ~12","lang":"python","description":"encoding_for_model() is safer than hardcoding encoding names — it handles model→encoding mapping automatically and stays correct as OpenAI adds new models. Token counts for chat completions must add per-message overhead (3 tokens per message) to get accurate billing estimates."},"warnings":[{"fix":"Use tiktoken.encoding_for_model('gpt-4o') to look up by model name, or pass the correct encoding name to get_encoding().","message":"get_encoding() takes an encoding name, NOT a model name. tiktoken.get_encoding('gpt-4o') raises KeyError. tiktoken.get_encoding('cl100k_base') is correct. This is the single most common error for new users.","severity":"breaking","affected_versions":"all"},{"fix":"Always use tiktoken.encoding_for_model(model_name) to get the correct encoding. Never hardcode cl100k_base as a universal default.","message":"gpt-4o and all o-series models (o1, o3, o4-mini) use o200k_base, NOT cl100k_base. Code that hardcodes cl100k_base for all OpenAI models silently produces wrong token counts for newer models — undercounting or overcounting by up to 10-15% depending on content.","severity":"breaking","affected_versions":"all"},{"fix":"Pre-warm the cache in a networked environment: tiktoken.get_encoding('o200k_base') and tiktoken.get_encoding('cl100k_base'). Set TIKTOKEN_CACHE_DIR to a writable path. Vocab files are then reused from disk.","message":"On first use, tiktoken downloads encoding vocab files (~1MB each) from OpenAI's CDN. In firewalled/offline environments this silently hangs or raises a connection error, not an ImportError.","severity":"gotcha","affected_versions":"all"},{"fix":"Use the OpenAI Cookbook's num_tokens_from_messages() pattern which adds per-message overhead. Don't use raw encode() length for chat token budgeting.","message":"Token count from enc.encode(text) counts tokens in raw text only. Chat completions add 3 overhead tokens per message and 3 tokens for the assistant reply primer. Omitting this overhead causes off-by-N errors in context window management.","severity":"gotcha","affected_versions":"all"},{"fix":"Install Rust via rustup (https://rustup.rs) before pip install if on an unsupported platform. Or use a Docker image with tiktoken pre-installed.","message":"Building from source requires Rust. On platforms or Python versions without a pre-built wheel, pip install tiktoken triggers a Rust compile. If Rust is not installed, the install fails with a cryptic error about 'cargo' not found.","severity":"gotcha","affected_versions":"all"},{"fix":"enc.encode('<|endoftext|>', allowed_special={'<|endoftext|>'}) or enc.encode(text, allowed_special='all') to allow all special tokens.","message":"Special tokens like <|endoftext|> are not encoded by default. enc.encode('<|endoftext|>') raises ValueError. Must explicitly allow them.","severity":"gotcha","affected_versions":"all"}],"env_vars":null,"search_vec":"'-6':21 '3':20 '4o':67 'api':32 'bpe':3,61 'build':41 'built':48 'call':30 'cl100k':68 'compar':25 'compil':39 'comput':35 'context':71 'context-window':70 'count':12,64 'encode/decode':15 'fast':2 'faster':23 'gpt':66 'gpt-4o':65 'import':53 'local':34 'model':19 'name':51,54 'o200k':69 'openai':6,18,59 'packag':50 'platform':44 'pre':47 'pre-built':46 'pure':33 'python':26 'requir':36 'rust':9,38 'text':16 'tiktoken':1,57,58 'time':42 'token':4,13,27,60,63 'token-count':62 'use':10 'wheel':49 'window':72 'without':45 'written':7 'x':22","created_at":"2026-03-16T04:39:09.011572+00:00","updated_at":"2026-04-16T22:57:03.713840+00:00","problems":[{"fix":"pip install tiktoken","cause":"The 'tiktoken' package is not installed in the Python environment where the script is being executed.","error":"ModuleNotFoundError: No module named 'tiktoken'"},{"fix":"Ensure you have the Rust toolchain installed (e.g., via 'rustup' for most platforms) or essential build tools (like 'build-essential' on Linux, Xcode Command Line Tools on macOS) before running `pip install tiktoken`.","cause":"tiktoken requires a Rust compiler to build from source if a pre-built wheel isn't available for your specific platform and Python version; the Rust toolchain or necessary system build tools are missing.","error":"Failed building wheel for tiktoken"},{"fix":"Verify the model name against OpenAI's documentation (e.g., 'gpt-4', 'gpt-3.5-turbo') and ensure your `tiktoken` library is up-to-date by running `pip install --upgrade tiktoken`.","cause":"The model name provided to `tiktoken.encoding_for_model()` is either incorrect, misspelled, or corresponds to a newer model that is not yet supported by your installed tiktoken version.","error":"KeyError: This model is not supported"}],"ecosystem":"pypi","meta_description":null,"install_score":85,"quickstart_score":80,"quickstart_tag":"verified","pypi_latest":"0.14.0","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/openai/tiktoken","docs":null,"changelog":"https://github.com/openai/tiktoken/blob/main/CHANGELOG.md","pypi":"https://pypi.org/project/tiktoken/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["llm-agents","ai-ml"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-27","last_verified":"2026-08-26","next_check":"2026-07-27","install_tag":"verified"}}