{"id":8655,"library":"snakemake-storage-plugin-s3","title":"Snakemake S3 Storage Plugin","description":"The `snakemake-storage-plugin-s3` library provides a Snakemake storage plugin for interacting with S3-compatible object storage systems, including AWS S3, MinIO, and others. It allows Snakemake workflows to read and write files directly from/to S3 buckets using `s3://` URIs. The project is actively maintained with frequent bug fix releases, typically every few months, ensuring compatibility and robustness.","status":"active","version":"0.3.6","language":"python","source_language":"en","source_url":"https://github.com/snakemake/snakemake-storage-plugin-s3","tags":["snakemake","s3","storage","aws","minio","cloud"],"install":[{"cmd":"pip install snakemake-storage-plugin-s3","lang":"bash","label":"Install plugin"}],"dependencies":[{"reason":"This is a plugin for Snakemake and requires Snakemake to be installed.","package":"snakemake","optional":false},{"reason":"Used for interacting with the S3 API.","package":"boto3","optional":false},{"reason":"Provides the necessary interface for Snakemake storage plugins.","package":"snakemake-interface-storage-plugins","optional":false}],"imports":[{"note":"You do not typically import classes directly from this library into your Python code. Snakemake discovers and utilizes the plugin automatically when it's installed in the same environment.","symbol":"S3Storage","correct":"This plugin is automatically discovered by Snakemake when installed, and is used via 's3://' URIs in your Snakefile, not by direct Python import of a specific class for usage."}],"quickstart":{"code":"# file: Snakefile\n\n# Set S3 credentials via environment variables for a runnable example\n# export AWS_ACCESS_KEY_ID=\"YOUR_ACCESS_KEY\"\n# export AWS_SECRET_ACCESS_KEY=\"YOUR_SECRET_KEY\"\n# export AWS_ENDPOINT_URL=\"http://localhost:9000\" # Optional, for MinIO or other S3-compatible storages\n\nrule all:\n    input: \"s3://my-test-bucket/output.txt\"\n\nrule generate_s3_file:\n    output: \"s3://my-test-bucket/output.txt\"\n    shell:\n        \"echo 'Hello Snakemake S3!' > {output}\"\n\n# To run this example:\n# 1. Ensure snakemake and snakemake-storage-plugin-s3 are installed.\n# 2. Set your S3 credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, etc.)\n#    or AWS_ENDPOINT_URL for compatible S3 services.\n# 3. Create 'my-test-bucket' in your S3 service.\n# 4. Navigate to the directory containing this Snakefile and run: snakemake","lang":"python","description":"This quickstart demonstrates how to use the S3 storage plugin in a `Snakefile`. Snakemake automatically recognizes `s3://` prefixes for input and output files when the plugin is installed. Ensure your S3 credentials are configured in your environment or via standard `boto3` configuration methods (e.g., `~/.aws/credentials`). For S3-compatible services like MinIO, `AWS_ENDPOINT_URL` is typically required."},"warnings":[{"fix":"Ensure all S3 object paths used in your Snakefile are absolute or relative paths without `..` segments. For example, use `s3://bucket/dir/file.txt` instead of `s3://bucket/dir/../file.txt`.","message":"S3 object keys cannot contain `..` (double-dot) for path traversal. The plugin attempts to normalize paths, but directly using `..` in S3 URIs will lead to errors.","severity":"gotcha","affected_versions":"All versions prior to 0.3.5, and potentially current if complex paths are not normalized correctly."},{"fix":"Upgrade to version 0.3.3 or newer, which includes a workaround for this `boto3` behavior. If issues persist, consult `boto3` documentation or your S3 provider for specific configuration requirements regarding checksums.","message":"When using non-AWS S3 compatible services (e.g., MinIO), `boto3` might automatically attach checksums to uploaded files which can cause issues or unexpected behavior.","severity":"gotcha","affected_versions":"All versions prior to 0.3.3."},{"fix":"Always keep the `snakemake-storage-plugin-s3` library updated to its latest version to ensure compatibility with recent Snakemake releases and the `snakemake-interface-storage-plugins`.","message":"Snakemake's storage plugin interface versioning requires the plugin to be compatible. If Snakemake is updated to a new major interface version, the S3 plugin might need an update as well.","severity":"gotcha","affected_versions":"Any version of the plugin that is not aligned with the Snakemake core's storage interface version (e.g., plugin 0.3.2 adapted to interface 4.0)."},{"fix":"Upgrade to version 0.3.4 or newer to fix issues with nested directory retrieval and `FileExists` errors when getting directories.","message":"In earlier versions, nested directories were not always retrieved correctly, leading to missing files or `FileNotFoundError`.","severity":"breaking","affected_versions":"Versions prior to 0.2.12 (for nested directories) and 0.3.4 (for `FileExists` bug)."}],"env_vars":null,"search_vec":"'activ':51 'allow':33 'aw':27,69 'bucket':44 'bug':55 'cloud':71 'compat':22,63 'direct':41 'ensur':62 'everi':59 'file':40 'fix':56 'frequent':54 'from/to':42 'includ':26 'interact':18 'librari':11 'maintain':52 'minio':29,70 'month':61 'object':23 'other':31 'plugin':4,9,16 'project':49 'provid':12 'read':37 'releas':57 'robust':65 's3':2,10,21,28,43,46,67 's3-compatible':20 'snakemak':1,7,14,34,66 'snakemake-storage-plugin-s3':6 'storag':3,8,15,24,68 'system':25 'typic':58 'uri':47 'use':45 'workflow':35 'write':39","created_at":"2026-04-16T17:03:18.453121+00:00","updated_at":"2026-04-16T17:03:18.453121+00:00","problems":{"verify_error":"no import statement found"},"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.3.6","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/snakemake/snakemake-storage-plugin-s3","docs":"https://snakemake.github.io/snakemake-plugin-catalog/plugins/storage/s3.html","changelog":null,"pypi":"https://pypi.org/project/snakemake-storage-plugin-s3/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["aws","data","devops","workflow","http-networking"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"skip","verified_at":"2026-06-28","last_verified":"2026-06-28","next_check":"2026-07-05","install_tag":null}}