{"id":2616,"library":"ndjson","title":"NDJSON Decoder for Python","description":"The `ndjson` library for Python, currently at version `0.3.1`, provides a `JsonDecoder` and `JsonEncoder` for newline-delimited JSON (NDJSON), also known as JSON Lines. It offers a familiar interface similar to Python's built-in `json` module, enabling efficient reading and writing of NDJSON data to and from file-like objects and strings. This lightweight library has no external dependencies and is particularly useful for processing large datasets or streaming applications where each line represents a complete, independent JSON object. Although its last release was in 2020 and its PyPI status is 'Pre-Alpha', it is considered stable and functional for its stated purpose.","status":"active","version":"0.3.1","language":"python","source_language":"en","source_url":"https://github.com/rhgrant10/ndjson","tags":["json","ndjson","jsonlines","serialization","deserialization","stream","newline-delimited"],"install":[{"cmd":"pip install ndjson","lang":"bash","label":"Install stable version"}],"dependencies":[],"imports":[{"symbol":"ndjson","correct":"import ndjson"},{"symbol":"load","correct":"ndjson.load(file_object)"},{"symbol":"dump","correct":"ndjson.dump(data, file_object)"},{"symbol":"loads","correct":"ndjson.loads(string_data)"},{"symbol":"dumps","correct":"ndjson.dumps(data)"},{"symbol":"reader","correct":"ndjson.reader(file_object)"},{"symbol":"writer","correct":"ndjson.writer(file_object)"}],"quickstart":{"code":"import ndjson\nimport os\n\n# Example data\ndata_to_write = [\n    {\"name\": \"Alice\", \"age\": 30, \"city\": \"New York\"},\n    {\"name\": \"Bob\", \"age\": 24, \"city\": \"San Francisco\"},\n    {\"name\": \"Charlie\", \"age\": 35, \"city\": \"London\"}\n]\n\nfile_path = \"example.ndjson\"\n\n# --- Writing NDJSON to a file ---\nwith open(file_path, 'w', encoding='utf-8') as f:\n    # Using ndjson.dump for a list of objects\n    ndjson.dump(data_to_write, f)\nprint(f\"Data written to {file_path} using ndjson.dump\")\n\n# Alternatively, using ndjson.writer for streaming individual rows\nfile_path_writer = \"example_writer.ndjson\"\nwith open(file_path_writer, 'w', encoding='utf-8') as f:\n    writer = ndjson.writer(f)\n    for record in data_to_write:\n        writer.writerow(record)\nprint(f\"Data written to {file_path_writer} using ndjson.writer\")\n\n# --- Reading NDJSON from a file ---\nread_data_dump = []\nwith open(file_path, 'r', encoding='utf-8') as f:\n    # Using ndjson.load for reading all objects from a file\n    read_data_dump = ndjson.load(f)\nprint(f\"\\nData read from {file_path} (ndjson.load):\\n{read_data_dump}\")\n\n# Alternatively, using ndjson.reader for streaming individual rows\nread_data_reader = []\nwith open(file_path_writer, 'r', encoding='utf-8') as f:\n    reader = ndjson.reader(f)\n    for row in reader:\n        read_data_reader.append(row)\nprint(f\"\\nData read from {file_path_writer} (ndjson.reader):\\n{read_data_reader}\")\n\n# Clean up created files\nos.remove(file_path)\nos.remove(file_path_writer)\n","lang":"python","description":"This quickstart demonstrates how to write and read NDJSON data using the `ndjson` library's `dump`/`load` functions for bulk operations and `writer`/`reader` classes for streaming line-by-line processing, similar to Python's `csv` module. This is particularly efficient for large files, avoiding the need to load the entire dataset into memory."},"warnings":[{"fix":"Be aware of the PyPI status, but understand that the library is stable for its intended use cases. The version 0.3.1 has been consistent since 2020.","message":"The `ndjson` library's official PyPI status is \"2 - Pre-Alpha\", which might suggest instability or an experimental nature. However, the library has been stable since its 0.3.1 release in February 2020 and \"works as advertised\", making this status misleading for its current functional state.","severity":"gotcha","affected_versions":"0.1.0 - 0.3.1"},{"fix":"Use `ndjson.load(file_object)` or iterate with `ndjson.reader(file_object)` for stream-based parsing, which correctly handles newline-delimited JSON objects.","message":"Do not attempt to parse an entire NDJSON file using Python's built-in `json.load()` (e.g., `json.load(open('data.ndjson'))`). This will typically result in a `json.JSONDecodeError` because NDJSON files contain multiple top-level JSON objects, not a single one, or a `MemoryError` for very large files.","severity":"gotcha","affected_versions":"All versions (general Python usage)"},{"fix":"The `ndjson` library handles this correctly with `ndjson.dump()` and `ndjson.writer()`. When manually constructing NDJSON, always ensure one valid JSON object per line.","message":"When writing NDJSON, ensure each record is a valid, self-contained JSON object on a single line, terminated by a newline character (`\\n`). Do not wrap the entire set of objects in a JSON array (`[]`) or add commas between objects, as this violates the NDJSON format and will cause parsing issues.","severity":"gotcha","affected_versions":"All versions (general NDJSON format adherence)"},{"fix":"Evaluate if the current feature set meets your needs. For very large-scale or high-performance NDJSON processing, consider alternatives like `ijson` for incremental parsing or `orjson` for faster JSON operations, or libraries integrated with dataframes like `polars.read_ndjson`.","message":"The library's last release (`0.3.1`) was in February 2020. While its core functionality is stable and complete for handling NDJSON, users seeking active development, bug fixes beyond the existing scope, or new features might find the project inactive.","severity":"deprecated","affected_versions":"0.3.1 and earlier"},{"fix":"Always specify `encoding='utf-8'` when opening NDJSON files. If a BOM is present and causing issues, use `encoding='utf-8-sig'` to automatically strip it.","message":"NDJSON files are expected to be UTF-8 encoded. Parsing issues can occur with files saved with a Byte Order Mark (BOM) or mixed encodings. This is a common issue for many text-based file formats.","severity":"gotcha","affected_versions":"All versions (general file handling)"}],"env_vars":null,"search_vec":"'0.3.1':13 '2020':94 'alpha':102 'also':25 'although':88 'applic':78 'built':40 'built-in':39 'complet':84 'consid':105 'current':10 'data':51 'dataset':75 'decod':2 'delimit':22,121 'depend':67 'deseri':117 'effici':45 'enabl':44 'extern':66 'familiar':33 'file':56 'file-lik':55 'function':108 'independ':85 'interfac':34 'json':23,28,42,86,113 'jsondecod':16 'jsonencod':18 'jsonlin':115 'known':26 'larg':74 'last':90 'librari':7,63 'lightweight':62 'like':57 'line':29,81 'modul':43 'ndjson':1,6,24,50,114 'newlin':21,120 'newline-delimit':20,119 'object':58,87 'offer':31 'particular':70 'pre':101 'pre-alpha':100 'process':73 'provid':14 'purpos':112 'pypi':97 'python':4,9,37 'read':46 'releas':91 'repres':82 'serial':116 'similar':35 'stabl':106 'state':111 'status':98 'stream':77,118 'string':60 'use':71 'version':12 'write':48","created_at":"2026-04-11T01:35:36.329983+00:00","updated_at":"2026-04-16T17:20:40.004409+00:00","problems":[{"fix":"pip install ndjson","cause":"The 'ndjson' library has not been installed in the current Python environment.","error":"ModuleNotFoundError: No module named 'ndjson'"},{"fix":"Use `ndjson.load()` for file-like objects, or iterate through the file line by line, parsing each line as a separate JSON object using `ndjson.loads()`:\n\n```python\nimport ndjson\n\n# Using ndjson.load() for file-like objects\nwith open('data.ndjson', 'r', encoding='utf-8') as f:\n    data = ndjson.load(f)\n\n# Or for very large files, process line by line\ndef read_ndjson_lines(filepath):\n    with open(filepath, 'r', encoding='utf-8') as f:\n        for line in f:\n            line = line.strip()\n            if line: # Skip empty lines\n                yield ndjson.loads(line)\n\nfor record in read_ndjson_lines('data.ndjson'):\n    print(record)\n```","cause":"Attempting to load a large NDJSON file using Python's standard `json.load()` method, which tries to read the entire file into memory as a single JSON object, exceeding available RAM.","error":"Process finished with exit code 137 (interrupted by signal 9: SIGKILL)"},{"fix":"Ensure each line in your NDJSON file represents a complete and syntactically correct JSON object. When parsing, you can include error handling to skip or log malformed lines:\n\n```python\nimport ndjson\nimport json\n\ndef parse_ndjson_with_error_handling(filepath):\n    valid_records = []\n    with open(filepath, 'r', encoding='utf-8') as f:\n        for line_num, line in enumerate(f, 1):\n            line = line.strip()\n            if not line: # Skip empty lines\n                continue\n            try:\n                record = ndjson.loads(line)\n                valid_records.append(record)\n            except json.JSONDecodeError as e:\n                print(f\"Error on line {line_num}: {e} - Content: {line[:100]}\")\n    return valid_records\n\nrecords = parse_ndjson_with_error_handling('malformed.ndjson')\nprint(f\"Successfully parsed {len(records)} records.\")\n```","cause":"An individual line within the NDJSON file is not a valid JSON object, containing syntax errors such as missing values, unquoted keys, incorrect delimiters, or unescaped characters.","error":"json.JSONDecodeError: Expecting value:"},{"fix":"Always specify `encoding='utf-8'` when opening files for writing and `ensure_ascii=False` when using `ndjson.dump()` or `ndjson.dumps()` if your data contains non-ASCII characters:\n\n```python\nimport ndjson\n\ndata_with_unicode = [{'name': 'Cáceres', 'city': 'Spain'}, {'name': '京都', 'city': 'Japan'}]\n\n# When dumping to a string\nunicode_text = ndjson.dumps(data_with_unicode, ensure_ascii=False)\nprint(unicode_text)\n\n# When dumping to a file\nwith open('output_unicode.ndjson', 'w', encoding='utf-8') as f:\n    ndjson.dump(data_with_unicode, f, ensure_ascii=False)\n```","cause":"Attempting to dump or encode JSON data containing non-ASCII characters without explicitly specifying UTF-8 encoding, often occurring in Python 2 environments or when default encoding is not UTF-8.","error":"UnicodeEncodeError: 'ascii' codec can't encode characters"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.3.1","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/rhgrant10/ndjson","docs":null,"changelog":null,"pypi":"https://pypi.org/project/ndjson/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["serialization","data"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-28","next_check":"2026-07-28","install_tag":null}}