{"id":3738,"library":"pipelinewise-singer-python","title":"PipelineWise Singer Python Library","description":"This library is a fork of Singer's singer-python, specifically tailored for PipelineWise compatibility. It provides utilities for implementing the Singer.io data replication specification, enabling taps (data extractors) and targets (data loaders) to communicate using a standard JSON-based message format over stdout. The current version is 2.0.1, with releases occurring infrequently, typically driven by critical bug fixes or significant feature enhancements like performance improvements.","status":"maintenance","version":"2.0.1","language":"python","source_language":"en","source_url":"https://github.com/transferwise/pipelinewise-singer-python","tags":["singer.io","etl","data integration","pipelinewise","data pipeline","tap","target","json"],"install":[{"cmd":"pip install pipelinewise-singer-python","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Switched to orjson in v2.0.0 for faster JSON serialization/deserialization.","package":"orjson","optional":false},{"reason":"Used for retry logic, bumped to 1.11.1 in v2.0.0.","package":"backoff","optional":false},{"reason":"Used for timezone handling, bumped to latest in v2.0.0.","package":"pytz","optional":false}],"imports":[{"note":"The entire library's functionality is typically accessed via the top-level 'singer' module after import.","symbol":"singer","correct":"import singer"}],"quickstart":{"code":"import singer\nimport sys\nimport json\n\n# Define a simple schema for demonstration\nschema = {\n    'properties': {\n        'id': {'type': 'integer', 'key': True},\n        'name': {'type': 'string'},\n        'value': {'type': 'number'}\n    }\n}\n\n# Write the schema message\nsinger.write_schema('my_stream', schema, ['id'])\n\n# Write some record messages\nrecords = [\n    {'id': 1, 'name': 'Item A', 'value': 100.5},\n    {'id': 2, 'name': 'Item B', 'value': 200.0},\n    {'id': 3, 'name': 'Item C', 'value': 150.75}\n]\n\nfor record in records:\n    singer.write_record('my_stream', record)\n\n# Write a state message (optional, but good practice for incremental processing)\nsinger.write_state({'last_processed_id': records[-1]['id']})\n\nprint(\"\\n--- Output captured (simulated stdout) ---\")\n# For demonstration, manually capture output to show what 'singer' writes\n# In a real Singer pipeline, this output goes to stdout.","lang":"python","description":"This quickstart demonstrates basic usage of the `pipelinewise-singer-python` library to emit Singer.io compliant messages (schema, record, state) to standard output. These messages can then be consumed by a Singer.io target. The example defines a simple schema and writes three records, followed by a state message."},"warnings":[{"fix":"Review custom JSON serialization/deserialization logic in your application. Test thoroughly after upgrading to ensure compatibility with `orjson`. Consult `orjson` documentation for any behavioral differences.","message":"Version 2.0.0 replaced the standard `json` library with `orjson` for improved performance. While generally a drop-in replacement, applications with custom JSON handling or those relying on specific `json` library behaviors not supported by `orjson` (e.g., certain `json.dumps` parameters) may experience unexpected issues.","severity":"breaking","affected_versions":">=2.0.0"},{"fix":"Implement a logging configuration in your application that utilizes `pipelinewise-singer-python`. Refer to Python's `logging` module documentation or set the `LOGGING_CONF_FILE` environment variable to a valid logging configuration file path.","message":"The library does not provide a default logging configuration. Users are responsible for setting up their own logging using standard Python `logging` module practices. If no logging is configured, log messages from the library might not appear or might go to stderr without proper formatting. An environment variable `LOGGING_CONF_FILE` can be used to point to a logging configuration file.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure all taps and targets in your Singer.io pipeline are updated to versions that explicitly support the `BATCH` message type, especially if you intend to leverage its performance benefits. Check component documentation for `BATCH` support.","message":"The `BATCH` message type, introduced in `v1.2.0` and enhanced in `v1.3.0` (with `time_extracted`), allows for more efficient data transfer. However, older taps or targets in a Singer.io pipeline might not fully support this message type, leading to compatibility issues or data processing failures if not all components are updated.","severity":"gotcha","affected_versions":"<1.2.0 (for pipelines not supporting BATCH)"}],"env_vars":null,"search_vec":"'2.0.1':55 'base':46 'bug':64 'communic':40 'compat':20 'critic':63 'current':52 'data':28,33,37,75,78 'driven':61 'enabl':31 'enhanc':69 'etl':74 'extractor':34 'featur':68 'fix':65 'fork':9 'format':48 'implement':25 'improv':72 'infrequ':59 'integr':76 'json':45,82 'json-bas':44 'librari':4,6 'like':70 'loader':38 'messag':47 'occur':58 'perform':71 'pipelin':79 'pipelinewis':1,19,77 'provid':22 'python':3,15 'releas':57 'replic':29 'signific':67 'singer':2,11,14 'singer-python':13 'singer.io':27,73 'specif':16,30 'standard':43 'stdout':50 'tailor':17 'tap':32,80 'target':36,81 'typic':60 'use':41 'util':23 'version':53","created_at":"2026-04-11T17:42:36.714809+00:00","updated_at":"2026-04-16T18:02:34.979083+00:00","problems":[{"fix":"Ensure the virtual environment is activated and use `import singer` in your Python code. If running a PipelineWise component, ensure PipelineWise is correctly installed and configured, as it manages the virtual environments for taps and targets.","cause":"The `pipelinewise-singer-python` library is installed as `pipelinewise-singer-python` via pip, but its core module for import is simply `singer`. This error occurs when a Python script tries to import `pipelinewise_singer_python` directly instead of `singer` or if the virtual environment where it's installed isn't active.","error":"ModuleNotFoundError: No module named 'singer'"},{"fix":"Verify that your configuration and state JSON files exist, are valid, and contain the expected data structure. Ensure they are correctly passed to the Singer tap or target. For PipelineWise, use `pipelinewise import --dir .` to generate the necessary JSON files from your YAML configurations.","cause":"This error often indicates that a configuration object (like `properties` or `state` in a Singer tap or target) is `None` when the code expects it to be a dictionary and tries to call the `.get()` method on it. This typically happens when a JSON configuration file (e.g., `config.json` or `state.json`) is missing, empty, or malformed, causing `utils.load_json()` to return `None`.","error":"AttributeError: 'NoneType' object has no attribute 'get'"},{"fix":"Review the generated Singer `catalog.json` and `schema` messages for the affected stream to ensure that all expected properties, especially 'type', are correctly defined. If using a custom tap, verify that it's emitting valid Singer schema messages. Ensure that all components (taps and targets) are compatible with the Singer specification they are expected to implement.","cause":"This `KeyError` typically arises when the `pipelinewise-singer-python` library or a component (like a target) attempts to access the 'type' key within a dictionary representing a schema property or a record, but the key is missing from that dictionary. This often points to an invalid or incomplete Singer schema being processed, especially when dealing with nested structures or when schema properties are empty.","error":"KeyError: 'type'"},{"fix":"Implement a logging configuration in your application using Python's `logging` module. Alternatively, set the `LOGGING_CONF_FILE` environment variable to the path of a valid logging configuration file (e.g., a `.conf` file).","cause":"By default, `pipelinewise-singer-python` does not set up a predefined logging configuration. If the calling application or PipelineWise itself doesn't configure Python's `logging` module, then messages from the library will not be formatted or directed to specific outputs, often resulting in them being silently dropped or printed unformatted to standard error.","error":"Log messages from the library might not appear or might go to stderr without proper formatting."}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"3.0.2","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/transferwise/pipelinewise-singer-python","docs":null,"changelog":null,"pypi":"https://pypi.org/project/pipelinewise-singer-python/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","workflow"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-29","next_check":"2026-07-28","install_tag":null}}