{"id":242,"library":"charset-normalizer","title":"Charset Normalizer","description":"Charset-normalizer is a truly universal charset encoding detector for Python. It detects the encoding of raw bytes/files using a heuristic, non-training-based approach and can optionally identify the spoken language of the content. All IANA character set names supported by CPython codecs are supported. The library also ships a `normalizer` CLI tool and a drop-in `detect()` shim for Chardet migration. Current version is 3.4.6 (released March 2026); releases follow Semantic Versioning with frequent minor/patch cadence.","status":"active","version":"3.4.6","language":"python","source_language":"en","source_url":"https://github.com/jawah/charset_normalizer","tags":["encoding","charset","detection","chardet","unicode","text","normalization","i18n"],"install":[{"cmd":"pip install charset-normalizer","lang":"bash","label":"PyPI (pure Python)"},{"cmd":"pip install charset-normalizer -U","lang":"bash","label":"Upgrade to latest"}],"dependencies":[],"imports":[{"note":"Primary API for detecting encoding of a bytes/bytearray object. Returns a CharsetMatches container.","symbol":"from_bytes","correct":"from charset_normalizer import from_bytes"},{"note":"Primary API for detecting encoding of a file on disk; accepts str, bytes, or os.PathLike.","symbol":"from_path","correct":"from charset_normalizer import from_path"},{"note":"Primary API for detecting encoding from an already-open binary file pointer. Does NOT close the file pointer.","symbol":"from_fp","correct":"from charset_normalizer import from_fp"},{"note":"Legacy Chardet-compatible shim. Officially deprecated in favour of from_bytes; not planned for removal. Returns a dict with 'encoding', 'confidence', 'language'.","wrong":"import chardet; chardet.detect(...)","symbol":"detect","correct":"from charset_normalizer import detect"},{"note":"Utility to detect whether bytes/path/fp point to binary (non-text) content. Added in 3.3.x.","symbol":"is_binary","correct":"from charset_normalizer import is_binary"},{"note":"Class was renamed from CharsetNormalizerMatches to CharsetMatches in 3.0. The old alias was removed.","wrong":"from charset_normalizer import CharsetNormalizerMatches","symbol":"CharsetMatches","correct":"from charset_normalizer.models import CharsetMatches"},{"note":"Renamed from CharsetNormalizerMatch in 3.0. Old alias removed.","wrong":"from charset_normalizer import CharsetNormalizerMatch","symbol":"CharsetMatch","correct":"from charset_normalizer.models import CharsetMatch"}],"quickstart":{"code":"from charset_normalizer import from_bytes, from_path, detect\n\n# --- from raw bytes ---\nraw = b'\\xff\\xfe' + 'Hello, world!'.encode('utf-16-le')\nresults = from_bytes(raw)\nbest = results.best()\nif best is not None:\n    print('Encoding:', best.encoding)        # e.g. 'utf_16'\n    print('Language:', best.language)        # e.g. 'English' or ''\n    print('Decoded :', str(best))            # decoded unicode string\nelse:\n    print('Could not detect encoding (possibly binary data)')\n\n# --- from a file path ---\n# results2 = from_path('./data/sample.txt')\n# print(str(results2.best()))\n\n# --- Chardet-compatible legacy shim (deprecated but stable) ---\nresult = detect(raw)\nprint(result)  # {'encoding': 'UTF-16', 'confidence': 1.0, 'language': ''}\nif result['encoding']:\n    decoded = raw.decode(result['encoding'])\n    print('Legacy decoded:', decoded)\n","lang":"python","description":"Detect encoding of raw bytes, decode the content, and use the Chardet-compatible legacy shim."},"warnings":[{"fix":"Replace with CharsetMatch and CharsetMatches imported from charset_normalizer.models, or use the top-level from_bytes/from_path functions directly.","message":"Class aliases CharsetNormalizerMatch, CharsetNormalizerMatches, CharsetDetector, and CharsetDoctor were removed in 3.0. Code referencing these names will raise ImportError or AttributeError.","severity":"breaking","affected_versions":"<3.0"},{"fix":"Pin charset-normalizer<3.1 for Python 3.6, or upgrade the Python interpreter.","message":"Python 3.6 support was dropped in 3.1.0, and Python 3.5 support was dropped in 2.1.0. Installing 3.x on Python 3.6 is unsupported.","severity":"breaking","affected_versions":"<3.1 for Python 3.6; <2.1 for Python 3.5"},{"fix":"Migrate to from_bytes(...).best() for new code. Check best() for None before calling str() or accessing .encoding.","message":"detect() is the legacy Chardet-compatible shim and is officially deprecated. It also lowers confidence automatically for small byte samples (3.4.3+), so results on short inputs may differ from Chardet.","severity":"gotcha","affected_versions":">=3.0"},{"fix":"Always pass the full byte sequence. Do not slice input for 'performance' — the library already samples internally (5 blocks of 512 bytes by default).","message":"Feeding truncated or incomplete multi-byte byte sequences (e.g. a partial UTF-16 or UTF-32 file) will likely produce incorrect or empty detection results. The library is not designed for streaming partial payloads.","severity":"gotcha","affected_versions":"all"},{"fix":"Use: result = from_bytes(raw).best(); text = str(result) if result is not None else ''","message":"from_bytes/from_path return a CharsetMatches container, not a string or a single result. Calling str() directly on the container gives unexpected output. Always call .best() first, then check for None.","severity":"gotcha","affected_versions":"all"},{"fix":"Always use: from charset_normalizer import ...","message":"The import name uses an underscore (charset_normalizer) but the PyPI/install name uses a hyphen (charset-normalizer). Using import charset-normalizer raises a SyntaxError.","severity":"gotcha","affected_versions":"all"},{"fix":"Do not import internal modules. Use only the public API: from_bytes, from_path, from_fp, detect, is_binary.","message":"Internal module charset_normalizer.assets was moved into charset_normalizer.constant in 3.3.x. Any code importing from charset_normalizer.assets directly will break on 3.3+.","severity":"deprecated","affected_versions":">=3.3"}],"env_vars":null,"search_vec":"'2026':75 '3.4.6':72 'also':53 'approach':29 'base':28 'bytes/files':21 'cadenc':83 'charact':42 'chardet':67,87 'charset':1,4,10,85 'charset-norm':3 'cli':57 'codec':48 'content':39 'cpython':47 'current':69 'detect':16,64,86 'detector':12 'drop':62 'drop-in':61 'encod':11,18,84 'follow':77 'frequent':81 'heurist':24 'i18n':91 'iana':41 'identifi':33 'languag':36 'librari':52 'march':74 'migrat':68 'minor/patch':82 'name':44 'non':26 'non-training-bas':25 'normal':2,5,56,90 'option':32 'python':14 'raw':20 'releas':73,76 'semant':78 'set':43 'shim':65 'ship':54 'spoken':35 'support':45,50 'text':89 'tool':58 'train':27 'truli':8 'unicod':88 'univers':9 'use':22 'version':70,79","created_at":"2026-03-27T17:05:50.384121+00:00","updated_at":"2026-04-16T01:55:54.323786+00:00","problems":[{"fix":"Reinstall the package cleanly using `pip install --force-reinstall charset-normalizer` or, if using conda, `conda install -c conda-forge charset-normalizer` after uninstalling any existing version.","cause":"This error typically indicates a corrupted or incomplete installation of `charset-normalizer`, often due to file shadowing, stale `__pycache__` files, or issues within specific build environments like PyInstaller.","error":"AttributeError: partially initialized module 'charset_normalizer' has no attribute 'md__mypyc' (most likely due to a circular import)"},{"fix":"Install the package using `pip install charset-normalizer` or `conda install charset-normalizer` depending on your environment.","cause":"The `charset-normalizer` package is not installed in the active Python environment or is not discoverable in the Python path.","error":"ModuleNotFoundError: No module named 'charset_normalizer'"},{"fix":"Ensure `charset-normalizer` is installed in an environment whose scripts directory is in your system's PATH, or run the tool using `python -m charset_normalizer`.","cause":"The `normalizer` CLI tool, which comes with the `charset-normalizer` library, is not found in your system's PATH or was not installed correctly.","error":"normalizer: command not found"},{"fix":"Cleanly uninstall both `charset-normalizer` and any directly dependent libraries (like `chardet` if present), then reinstall `charset-normalizer` and the dependent libraries to ensure compatible versions are used.","cause":"This usually points to a version incompatibility or a corrupted installation, often occurring when `charset-normalizer` is used alongside other libraries (like `transformers` or `chardet`) that expect a different internal structure or version.","error":"ImportError: cannot import name 'COMMON_SAFE_ASCII_CHARACTERS' from 'charset_normalizer.constant'"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"quickstart_score":80,"quickstart_tag":"verified","pypi_latest":"3.5.1","cli_name":"normalizer","cli_version":"Charset-Normalizer 3.4.7 - Python 3.11.15 - Unicode 14.0.0 - SpeedUp ON","type":"library","homepage":null,"github":"https://github.com/jawah/charset_normalizer","docs":"https://charset-normalizer.readthedocs.io/","changelog":"https://github.com/jawah/charset_normalizer/blob/master/CHANGELOG.md","pypi":"https://pypi.org/project/charset-normalizer/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["serialization","data"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-27","last_verified":"2026-08-27","next_check":"2026-07-27","install_tag":"verified"}}