{"id":5927,"library":"fastwarc","title":"FastWARC","description":"FastWARC is a high-performance Python library for parsing WARC (Web ARChive) files, written in C++/Cython. It supports WARC/1.0 and WARC/1.1 streams with GZip and LZ4 compression, offering significant speed improvements over pure Python alternatives like WARCIO. FastWARC is part of the ChatNoir Resiliparse toolkit and is currently at version 0.16.0, with active development.","status":"active","version":"0.16.0","language":"python","source_language":"en","source_url":"https://github.com/chatnoir-eu/chatnoir-resiliparse","tags":["WARC","web archives","parsing","performance","Cython","C++","data processing"],"install":[{"cmd":"pip install fastwarc","lang":"bash","label":"Standard installation"},{"cmd":"pip install fastwarc[fsspec]","lang":"bash","label":"With fsspec for remote filesystems"},{"cmd":"sudo apt install build-essential python3-dev zlib1g-dev liblz4-dev && pip install --no-binary fastwarc fastwarc","lang":"bash","label":"Build from source for optimal Linux performance"}],"dependencies":[{"reason":"Enables reading WARC files from remote filesystems and URLs.","package":"fsspec","optional":true}],"imports":[{"symbol":"ArchiveIterator","correct":"from fastwarc.warc import ArchiveIterator"},{"symbol":"WarcRecord","correct":"from fastwarc.warc import WarcRecord"},{"symbol":"WarcRecordType","correct":"from fastwarc.warc import WarcRecordType"}],"quickstart":{"code":"import os\nfrom fastwarc.warc import ArchiveIterator, WarcRecordType\n\n# Create a dummy WARC file for demonstration purposes\ndummy_warc_content = b'WARC/1.0\\r\\nWARC-Type: warcinfo\\r\\nWARC-Date: 2023-01-01T12:00:00Z\\r\\nWARC-Record-ID: <urn:uuid:example-warcinfo>\\r\\nContent-Length: 100\\r\\n\\r\\ninfo: This is a dummy WARC file created for FastWARC quickstart example.\\n123456789012345678901234567890\\r\\nWARC/1.0\\r\\nWARC-Type: response\\r\\nWARC-Date: 2023-01-01T12:00:01Z\\r\\nWARC-Record-ID: <urn:uuid:example-response>\\r\\nWARC-Target-URI: http://example.com/\\r\\nContent-Length: 77\\r\\n\\r\\nHTTP/1.1 200 OK\\r\\nContent-Type: text/plain\\r\\n\\r\\nHello, FastWARC!\\n'\nwith open('example.warc', 'wb') as f:\n    f.write(dummy_warc_content)\n\nwarc_path = 'example.warc'\n\nif not os.path.exists(warc_path):\n    print(f\"Error: WARC file '{warc_path}' not found. Please ensure it exists.\")\nelse:\n    try:\n        # Iterate over WARC records, parsing HTTP for response records\n        for record in ArchiveIterator(warc_path, parse_http=True):\n            if record.record_type == WarcRecordType.warcinfo:\n                print(f\"WARC Info Record ID: {record.record_id}\")\n                print(f\"Content: {record.reader.read().decode('utf-8').strip()}\")\n            elif record.record_type == WarcRecordType.response:\n                print(f\"\\nResponse Record URL: {record.url}\")\n                if record.http_headers:\n                    print(f\"HTTP Status: {record.http_headers.status_code}\")\n                print(f\"Payload: {record.reader.read().decode('utf-8').strip()}\")\n    except Exception as e:\n        print(f\"An error occurred during WARC processing: {e}\")\n    finally:\n        # Clean up the dummy WARC file\n        os.remove(warc_path)\n","lang":"python","description":"This quickstart demonstrates how to iterate through records in a WARC file using `ArchiveIterator`. It shows how to access record metadata like `record_id` and `url`, and how to read the content. For HTTP response records, it also shows how to access parsed HTTP headers. A dummy WARC file is created for the example to be runnable."},"warnings":[{"fix":"Review FastWARC documentation for API differences when migrating from WARCIO.","message":"FastWARC is not a drop-in replacement for WARCIO. Its API is inspired by WARCIO but designed for performance, meaning direct migration may require code adjustments.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Pass `strict_mode=False` to `ArchiveIterator` for more lenient parsing of non-compliant WARC files. Be aware this might affect how record boundaries are determined.","message":"Malformed WARC records (e.g., missing Content-Length, non-standard line endings) in archives like ClueWeb can cause parsing issues. By default, `strict_mode=True` which may lead to early termination.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Set `parse_http=False` in the `ArchiveIterator` constructor. You can parse HTTP headers later on a per-record basis using `record.parse_http()` if needed.","message":"Automatic HTTP parsing (`parse_http=True`) can incur a performance overhead. If you only need WARC metadata or raw content and not parsed HTTP headers, this can be skipped.","severity":"gotcha","affected_versions":"All versions"},{"fix":"For large records, ensure they fit into memory or set `consume=True` when calling digest verification methods if you do not need to preserve the stream contents for subsequent operations. This avoids creating a stream copy.","message":"Verifying record digests (e.g., `record.verify_block_digest()`) creates an in-memory copy of the remaining record stream to preserve its contents for further processing. This can consume significant memory for very large records.","severity":"gotcha","affected_versions":"All versions"},{"fix":"If you require ARC format compatibility, you will need to use a different library such as WARCIO.","message":"FastWARC explicitly does not support the legacy ARC (Archive Record) format for simplicity and performance reasons.","severity":"breaking","affected_versions":"All versions"},{"fix":"For the best performance on Linux, it is recommended to build FastWARC from source by installing build dependencies (`build-essential`, `python3-dev`, `zlib1g-dev`, `liblz4-dev`) and then using `pip install --no-binary fastwarc fastwarc`.","message":"Pre-built Linux binaries are compiled on an older `manylinux` base system for compatibility, which may not offer optimal performance on modern systems.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"search_vec":"'/cython':19 '0.16.0':54 'activ':56 'altern':38 'archiv':14,60 'c':18,64 'chatnoir':46 'compress':30 'current':51 'cython':63 'data':65 'develop':57 'fastwarc':1,2,41 'file':15 'gzip':27 'high':6 'high-perform':5 'improv':34 'librari':9 'like':39 'lz4':29 'offer':31 'pars':11,61 'part':43 'perform':7,62 'process':66 'pure':36 'python':8,37 'resilipars':47 'signific':32 'speed':33 'stream':25 'support':21 'toolkit':48 'version':53 'warc':12,58 'warc/1.0':22 'warc/1.1':24 'warcio':40 'web':13,59 'written':16","created_at":"2026-04-14T18:34:28.703511+00:00","updated_at":"2026-04-16T14:59:27.661235+00:00","problems":[{"fix":"Install fastwarc using pip. If pre-built binaries are not available or fail, ensure build dependencies are installed for your operating system (e.g., `sudo apt install build-essential python3-dev zlib1g-dev liblz4-dev` on Ubuntu) and then run `pip install --no-binary fastwarc fastwarc` to build from source.","cause":"The fastwarc library is not installed in the current Python environment, or its installation failed due to missing C/Cython build dependencies.","error":"ModuleNotFoundError: No module named 'fastwarc'"},{"fix":"Replace `WarcReader` with `ArchiveIterator`. The correct import is `from fastwarc.warc import ArchiveIterator`.","cause":"Developers migrating from WARCIO (which has a `WarcReader` class) might attempt to import a non-existent class from `fastwarc`. FastWARC uses `ArchiveIterator` as its primary class for iterating WARC records.","error":"ImportError: cannot import name 'WarcReader' from 'fastwarc.warc'"},{"fix":"Access the record's payload stream using `record.reader` instead of `record.raw_stream`. For example, `payload_data = record.reader.read()` will read the body.","cause":"This error occurs when attempting to access the `raw_stream` attribute on a `fastwarc.warc.WarcRecord` object, an attribute common in WARCIO's `ArcWarcRecord` but not present in FastWARC. FastWARC uses `record.reader` to access the payload stream.","error":"AttributeError: 'WarcRecord' object has no attribute 'raw_stream'"},{"fix":"Implement robust error handling by wrapping WARC processing in a `try...except fastwarc.stream_io.FastWARCError` block. For problematic WARC files (e.g., ClueWeb), consider initializing `ArchiveIterator` with `strict_mode=False` or `parse_http=False` to handle non-standard formats.","cause":"This is a generic error raised by FastWARC indicating an issue during stream processing, often due to malformed WARC records, unexpected end-of-stream, or data corruption within the WARC file.","error":"fastwarc.stream_io.FastWARCError: FastWARC stream error"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"1.0.7","cli_name":"","cli_version":null,"type":"library","homepage":"https://resiliparse.chatnoir.eu","github":"https://github.com/chatnoir-eu/chatnoir-resiliparse","docs":"https://resiliparse.chatnoir.eu","changelog":null,"pypi":"https://pypi.org/project/fastwarc/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","serialization"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-06-28","next_check":"2026-07-28","install_tag":null}}