{"id":783,"library":"msgspec","title":"msgspec","description":"msgspec is a fast serialization and validation library, with builtin support for JSON, MessagePack, YAML, and TOML. It features high-performance encoders/decoders, zero-cost schema validation using Python type annotations, and a speedy `Struct` type. The library is actively maintained with frequent releases.","status":"active","version":"0.20.0","language":"python","source_language":"en","source_url":"https://github.com/jcrist/msgspec","tags":["serialization","validation","json","msgpack","yaml","toml","performance","structs","dataclasses-alternative"],"install":[{"cmd":"pip install msgspec","lang":"bash","label":"Basic Install"},{"cmd":"pip install msgspec[yaml,toml]","lang":"bash","label":"Install with YAML and TOML support"}],"dependencies":[{"reason":"Required for `msgspec.yaml` module functionality.","package":"PyYAML","optional":true},{"reason":"Required for `msgspec.toml` module functionality.","package":"tomli_w","optional":true}],"imports":[{"symbol":"Struct","correct":"from msgspec import Struct"},{"symbol":"field","correct":"from msgspec import field"},{"note":"Provides `encode`, `decode`, `Encoder`, `Decoder` for JSON.","symbol":"json","correct":"import msgspec.json"},{"note":"Provides `encode`, `decode`, `Encoder`, `Decoder` for MessagePack.","symbol":"msgpack","correct":"import msgspec.msgpack"},{"note":"Provides `encode`, `decode`, `Encoder`, `Decoder` for YAML (requires PyYAML).","symbol":"yaml","correct":"import msgspec.yaml"},{"note":"Provides `encode`, `decode`, `Encoder`, `Decoder` for TOML (requires tomli-w).","symbol":"toml","correct":"import msgspec.toml"}],"quickstart":{"code":"import msgspec\n\nclass User(msgspec.Struct):\n    name: str\n    age: int\n    email: str | None = None\n    groups: set[str] = msgspec.field(default_factory=set)\n\nalice = User(name=\"alice\", age=30, groups={\"admin\", \"dev\"})\nprint(f\"Original User: {alice}\")\n\n# Encode to JSON\njson_data = msgspec.json.encode(alice)\nprint(f\"Encoded JSON: {json_data.decode()}\")\n\n# Decode from JSON\ndecoded_alice = msgspec.json.decode(json_data, type=User)\nprint(f\"Decoded User: {decoded_alice}\")\n\n# Example of validation error\ntry:\n    msgspec.json.decode(b'{\"name\":\"bob\",\"age\":\"25\"}', type=User)\nexcept msgspec.ValidationError as e:\n    print(f\"Validation Error: {e}\")","lang":"python","description":"Defines a simple `User` struct, encodes it to JSON, and then decodes it back. It also demonstrates how `msgspec` handles type validation errors during decoding. Uses `msgspec.field(default_factory=set)` for mutable default values to prevent common Python footguns."},"warnings":[{"fix":"Upgrade Python to 3.9+.","message":"msgspec 0.19.0 dropped support for Python 3.8. Users on Python 3.8 or older must upgrade their Python version or stay on msgspec < 0.19.0.","severity":"breaking","affected_versions":">=0.19.0"},{"fix":"Review usage of `Encoder.encode_into` and ensure buffer management aligns with the new behavior, potentially pre-allocating larger buffers or handling re-allocation.","message":"In msgspec 0.19.0, the `encode_into` method (an advanced API) now expands the buffer if it's smaller than the offset, which is a breaking change for specific buffer management use cases.","severity":"breaking","affected_versions":">=0.19.0"},{"fix":"Replace calls to `from_builtins` with `msgspec.convert`.","message":"The `from_builtins` method was removed in msgspec 0.19.0. Users should now use `msgspec.convert` for similar functionality.","severity":"deprecated","affected_versions":">=0.19.0"},{"fix":"Only use `memoryview` for zero-copy scenarios when strict performance is needed and memory lifecycle is precisely controlled. For most cases, prefer `bytes` or `bytearray` to ensure the input buffer can be garbage collected.","message":"When using `msgspec.msgpack` to decode into `memoryview` objects, the original input message buffer will be kept in memory as long as the `memoryview` is alive. This can lead to unexpectedly high memory usage if not carefully managed, especially with large messages or long-lived `memoryview` instances.","severity":"gotcha","affected_versions":"All"},{"fix":"Always use `msgspec.field(default_factory=my_callable)` for mutable default values, where `my_callable` is a zero-argument function that returns a new mutable object (e.g., `list`, `set`, `dict`).","message":"Mutable default values (e.g., `list`, `set`, `dict`) on `msgspec.Struct` fields will be shared across all instances if not defined using `msgspec.field(default_factory=...)`. This is a common Python pitfall.","severity":"gotcha","affected_versions":"All"},{"fix":"Cache the result of `msgspec.structs.fields()` if it's called repeatedly for the same `Struct` type, or redesign code to minimize its usage in hot paths.","message":"The `msgspec.structs.fields()` function can be significantly slower (up to 20x) than `dataclasses.fields()` because it re-inspects type annotations on every invocation. Avoid frequent calls to `msgspec.structs.fields()` in performance-critical loops.","severity":"gotcha","affected_versions":"All"},{"fix":"Update `Struct` definitions and instantiation calls from `nogc=True` to `gc=False`.","message":"In msgspec 0.7.0, the `nogc` struct option was renamed to `gc`. To disable garbage collection for a Struct instance, you must now specify `gc=False` instead of `nogc=True`.","severity":"breaking","affected_versions":">=0.7.0"},{"fix":"For Python 3.9 and older, replace type hints like `email: str | None` with `email: typing.Optional[str]` or `email: typing.Union[str, None]`. Remember to import `typing` if not already done.","message":"Type hints using the `X | Y` syntax (e.g., `str | None`) for unions are only supported in Python 3.10 and later. When using `msgspec.Struct` on Python 3.9 or older, this syntax will raise a TypeError.","severity":"breaking","affected_versions":"All"},{"fix":"Ensure input data types precisely match the `msgspec.Struct` field annotations. If type coercion is required, implement custom hooks or preprocess the data before decoding.","message":"Msgspec performs strict type validation by default. It does not automatically coerce values from one primitive type to another (e.g., a string '123' will not be converted to an integer 123) during decoding. Attempting to decode input data where a field's type does not strictly match its schema definition will raise a `ValidationError`.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"search_vec":"'activ':42 'altern':57 'annot':33 'builtin':11 'cost':27 'dataclass':56 'dataclasses-altern':55 'encoders/decoders':24 'fast':5 'featur':20 'frequent':45 'high':22 'high-perform':21 'json':14,49 'librari':9,40 'maintain':43 'messagepack':15 'msgpack':50 'msgspec':1,2 'perform':23,53 'python':31 'releas':46 'schema':28 'serial':6,47 'speedi':36 'struct':37,54 'support':12 'toml':18,52 'type':32,38 'use':30 'valid':8,29,48 'yaml':16,51 'zero':26 'zero-cost':25","created_at":"2026-03-29T04:21:30.351136+00:00","updated_at":"2026-04-16T16:41:40.485353+00:00","problems":[{"fix":"Reorder the fields in your `msgspec.Struct` definition so that all required fields come before optional fields, or set `kw_only=True` in the `Struct` definition to make all fields keyword-only.\n\n```python\nimport msgspec\n\n# Fix 1: Reorder fields\nclass ValidOrder(msgspec.Struct):\n    a: str  # Required\n    b: int = 0  # Optional\n\n# Fix 2: Use kw_only=True\nclass KeywordOnly(msgspec.Struct, kw_only=True):\n    a: str = \"\" # Optional, but position doesn't matter for kw_only\n    b: int # Required\n```","cause":"Python function signature rules, which `msgspec.Struct` adheres to, require that all fields without a default value (required fields) must be defined before any fields with a default value (optional fields).","error":"TypeError: Required field 'b' cannot follow optional fields. Either reorder the struct fields, or set `kw_only=True` in the struct definition."},{"fix":"Ensure that the input data matches the type annotations specified in your `msgspec.Struct` or the `type` argument passed to the decoder. Inspect the input JSON/MessagePack and the `msgspec.Struct` definition to find the mismatch.\n\n```python\nimport msgspec\n\nclass User(msgspec.Struct):\n    name: str\n    groups: list[str] = msgspec.field(default_factory=list)\n\n# Correct input: groups contains only strings\nvalid_data = b'{\"name\":\"bob\",\"groups\":[\"devops\"]}'\nuser = msgspec.json.decode(valid_data, type=User)\nprint(user)\n\n# Original problematic input (assuming it contained an int where str was expected)\n# invalid_data = b'{\"name\":\"bob\",\"groups\":[\"devops\", 123]}'\n# try:\n#     msgspec.json.decode(invalid_data, type=User)\n# except msgspec.ValidationError as e:\n#     print(e)\n```","cause":"This error occurs during decoding when the input data does not conform to the expected Python type annotations defined in the `msgspec.Struct` (or other specified type). In this example, an integer was found where a string was expected within a list.","error":"msgspec.ValidationError: Expected `str`, got `int` - at `$.groups[0]`"},{"fix":"Convert unsupported third-party types to a supported Python native type (e.g., `float`, `int`, `list`, `dict`) before encoding with `msgspec`. For custom types, you can also provide an `enc_hook` to the encoder.\n\n```python\nimport msgspec\nimport numpy as np\n\n# Original problematic code:\n# mjson.encode(np.float64(1.0))\n\n# Fix 1: Convert to a native Python float\nvalue_np = np.float64(1.0)\nencoded_data = msgspec.json.encode(float(value_np))\nprint(encoded_data)\n\n# Fix 2: Use an enc_hook for custom handling\ndef numpy_encoder_hook(obj):\n    if isinstance(obj, np.ndarray):\n        return obj.tolist()\n    if isinstance(obj, np.generic):\n        return obj.item()\n    raise NotImplementedError\n\nencoder = msgspec.json.Encoder(enc_hook=numpy_encoder_hook)\nencoded_data_with_hook = encoder.encode(np.array([1.0, np.float64(2.0)]))\nprint(encoded_data_with_hook)\n```","cause":"`msgspec` does not natively support encoding all arbitrary third-party types, such as NumPy data types, directly. It expects standard Python types or `msgspec.Struct` instances.","error":"TypeError: Encoding objects of type numpy.float64 is unsupported."},{"fix":"If you need to modify the object, either define the `msgspec.Struct` without `frozen=True`, or create a new instance with the desired changes.\n\n```python\nimport msgspec\n\nclass Point(msgspec.Struct, frozen=True):\n    x: float\n    y: float\n\np = Point(1.0, 2.0)\n\n# Original problematic code:\n# p.x = 2.0\n\n# Fix: Create a new instance with updated values\np_new = Point(x=3.0, y=p.y) # Or use msgspec.structs.replace if you have many fields\nprint(p_new)\n\n# If mutability is desired, define the struct without frozen=True\nclass MutablePoint(msgspec.Struct):\n    x: float\n    y: float\nm_p = MutablePoint(1.0, 2.0)\nm_p.x = 3.0\nprint(m_p)\n```","cause":"This error occurs when attempting to modify an attribute of a `msgspec.Struct` instance that was defined with `frozen=True`. Frozen structs are immutable after initialization, preventing any attribute changes.","error":"AttributeError: immutable type: 'Point'"},{"fix":"Refactor your type annotations to avoid unions with multiple string-like types. If you need to handle both `str` and `bytes`, consider using a single, unambiguous type or process `bytes` separately (e.g., base64 encode/decode if using JSON) or use `msgspec.Raw` for manual handling.\n\n```python\nimport msgspec\nfrom typing import Union\n\n# Original problematic code:\n# class TestData(msgspec.Struct):\n#     content: Union[str, bytes]\n\n# Fix 1: Use a single, unambiguous type\nclass TestDataStr(msgspec.Struct):\n    content: str\n\nclass TestDataBytes(msgspec.Struct):\n    content: bytes\n\n# Fix 2: If you must handle both, define separate structs and use a Tagged Union\n# This allows msgspec to differentiate between them\nclass MyStrData(msgspec.Struct, tag='str_data'):\n    value: str\n\nclass MyBytesData(msgspec.Struct, tag='bytes_data'):\n    value: bytes\n\nclass Wrapper(msgspec.Struct):\n    data: Union[MyStrData, MyBytesData]\n\nencoder = msgspec.msgpack.Encoder()\ndecoder = msgspec.msgpack.Decoder(Wrapper)\n\nwrapped_str = Wrapper(MyStrData('hello'))\nencoded_str = encoder.encode(wrapped_str)\ndecoded_str = decoder.decode(encoded_str)\nprint(decoded_str)\n\nwrapped_bytes = Wrapper(MyBytesData(b'world'))\nencoded_bytes = encoder.encode(wrapped_bytes)\ndecoded_bytes = decoder.decode(encoded_bytes)\nprint(decoded_bytes)\n```","cause":"`msgspec` enforces a restriction that type unions cannot contain multiple 'string-like' types (e.g., `str` and `bytes`) to avoid ambiguity, especially when handling different serialization formats like JSON (which has only strings) and MessagePack (which distinguishes strings and binary data).","error":"TypeError: Type unions may not contain more than one str-like type (`str`, `Enum`, ...)"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"quickstart_score":70,"quickstart_tag":"verified","pypi_latest":"0.21.1","cli_name":"","cli_version":null,"type":"library","homepage":"https://jcristharif.com/msgspec/","github":"https://github.com/jcrist/msgspec","docs":null,"changelog":null,"pypi":"https://pypi.org/project/msgspec/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["serialization","data"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-06-28","next_check":"2026-07-28","install_tag":"verified"}}