{"id":5717,"library":"s5cmd","title":"s5cmd Python Distributions","description":"This project provides Python wheels for the high-performance `s5cmd` command-line tool, a utility written in Go for managing S3 and S3-compatible object storage systems. It focuses on speed and efficiency for bulk operations, parallel processing, and advanced filtering. The Python package ensures the `s5cmd` executable is available in the user's PATH after installation, allowing Python applications to invoke the CLI tool via subprocess. The current version is 0.3.3, with a release cadence tied to updates of the underlying `s5cmd` binary and build infrastructure improvements.","status":"active","version":"0.3.3","language":"python","source_language":"en","source_url":"https://github.com/jcfr/s5cmd-python-distributions","tags":["aws","s3","object storage","cli wrapper","performance","go"],"install":[{"cmd":"pip install s5cmd","lang":"bash","label":"Install `s5cmd` distribution package"}],"dependencies":[],"imports":[{"note":"The `s5cmd` package primarily installs the `s5cmd` command-line executable. Interaction from Python is typically done by invoking this executable using the `subprocess` module.","symbol":"subprocess","correct":"import subprocess"}],"quickstart":{"code":"import subprocess\nimport os\n\n# Ensure AWS credentials are configured (e.g., via environment variables or ~/.aws/credentials)\n# For example, using environment variables for demonstration:\n# os.environ['AWS_ACCESS_KEY_ID'] = os.environ.get('AWS_ACCESS_KEY_ID', 'YOUR_AWS_ACCESS_KEY_ID')\n# os.environ['AWS_SECRET_ACCESS_KEY'] = os.environ.get('AWS_SECRET_ACCESS_KEY', 'YOUR_AWS_SECRET_ACCESS_KEY')\n# os.environ['AWS_REGION'] = os.environ.get('AWS_REGION', 'us-east-1')\n\ntry:\n    # Verify s5cmd is installed and accessible in PATH\n    version_output = subprocess.run(['s5cmd', 'version'], capture_output=True, text=True, check=True)\n    print(f\"s5cmd version:\\n{version_output.stdout}\")\n\n    # Example: List objects in an S3 bucket (replace with a real bucket)\n    bucket_name = \"your-test-s5cmd-bucket\"\n    list_command = ['s5cmd', 'ls', f's3://{bucket_name}/']\n    list_result = subprocess.run(list_command, capture_output=True, text=True, check=True)\n    print(f\"\\nListing objects in s3://{bucket_name}/:\\n{list_result.stdout}\")\n\n    # Example: Create a dummy local file and upload it\n    local_file_name = \"hello_s5cmd.txt\"\n    with open(local_file_name, \"w\") as f:\n        f.write(\"Hello from s5cmd Python distribution!\")\n    \n    upload_command = ['s5cmd', 'cp', local_file_name, f's3://{bucket_name}/{local_file_name}']\n    upload_result = subprocess.run(upload_command, capture_output=True, text=True, check=True)\n    print(f\"\\nUploaded {local_file_name}:\\n{upload_result.stdout}\")\n\n    # Example: Download the file back\n    download_command = ['s5cmd', 'cp', f's3://{bucket_name}/{local_file_name}', f'./downloaded_{local_file_name}']\n    download_result = subprocess.run(download_command, capture_output=True, text=True, check=True)\n    print(f\"\\nDownloaded 'downloaded_{local_file_name}':\\n{download_result.stdout}\")\n    \n    # Clean up local files\n    os.remove(local_file_name)\n    os.remove(f'./downloaded_{local_file_name}')\n\n    # Note: For production, consider robust error handling and command construction.\n    # Ensure the bucket 'your-test-s5cmd-bucket' exists and credentials have write access.\n\nexcept FileNotFoundError:\n    print(\"Error: 's5cmd' command not found. Ensure it's installed and in your system's PATH.\")\nexcept subprocess.CalledProcessError as e:\n    print(f\"Error executing s5cmd command: {e}\")\n    print(f\"Stdout: {e.stdout}\")\n    print(f\"Stderr: {e.stderr}\")\nexcept Exception as e:\n    print(f\"An unexpected error occurred: {e}\")\n","lang":"python","description":"This quickstart demonstrates how to check for `s5cmd` and execute basic S3 operations (list, upload, download) using Python's `subprocess` module. It assumes `s5cmd` is correctly installed via `pip install s5cmd` and that AWS credentials are configured in the environment or standard locations for `s5cmd` to access S3."},"warnings":[{"fix":"If you need a direct Python API, install `s5cmdpy` (`pip install s5cmdpy`) instead. If you intend to call the command-line tool, use `subprocess` as shown in the quickstart.","message":"This `s5cmd` package (from `ImagingDataCommons/s5cmd-python-distributions`) primarily distributes the `s5cmd` Go binary and makes it executable. It does NOT provide a direct Python API for S3 interactions. For a Pythonic wrapper that offers `S5CmdRunner` class and direct function calls, consider `s5cmdpy` (from `trojblue/s5cmd-python`).","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure your environment or `~/.aws/credentials` file is correctly set up with valid AWS credentials and region information before running `s5cmd` commands.","message":"`s5cmd` (the underlying Go tool) relies on standard AWS credential configuration (e.g., `~/.aws/credentials`, environment variables like `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`) to authenticate with S3. Without proper configuration, commands will fail.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Review the `s5cmd` official documentation for recommended `--numworkers`, `-uw` (upload workers), and `-dw` (download workers) flags and adjust them based on your network, S3 bucket, and file characteristics to maximize throughput. Example: `s5cmd -uw 32 -dw 16 cp ...`","message":"Achieving the advertised high performance of `s5cmd` often requires tuning concurrency parameters (e.g., `--numworkers`, `-uw`, `-dw`) which default to conservative values. Suboptimal settings can lead to significantly slower transfers.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"search_vec":"'0.3.3':78 'advanc':46 'allow':64 'applic':66 'avail':56 'aw':95 'binari':90 'build':92 'bulk':41 'cadenc':82 'cli':70,99 'command':16 'command-lin':15 'compat':30 'current':75 'distribut':3 'effici':39 'ensur':51 'execut':54 'filter':47 'focus':35 'go':23,102 'high':12 'high-perform':11 'improv':94 'infrastructur':93 'instal':63 'invok':68 'line':17 'manag':25 'object':31,97 'oper':42 'packag':50 'parallel':43 'path':61 'perform':13,101 'process':44 'project':5 'provid':6 'python':2,7,49,65 'releas':81 's3':26,29,96 's3-compatible':28 's5cmd':1,14,53,89 'speed':37 'storag':32,98 'subprocess':73 'system':33 'tie':83 'tool':18,71 'under':88 'updat':85 'user':59 'util':20 'version':76 'via':72 'wheel':8 'wrapper':100 'written':21","created_at":"2026-04-14T03:40:43.042362+00:00","updated_at":"2026-04-16T21:16:19.427590+00:00","problems":[{"fix":"Ensure the `s5cmd` Python package is installed (`pip install s5cmd`) and use `s5cmd.get_s5cmd_path()` to reliably locate the executable path for `subprocess.run()`.","cause":"The Python `subprocess` module cannot find the `s5cmd` executable in the system's PATH.","error":"FileNotFoundError: [Errno 2] No such file or directory: 's5cmd'"},{"fix":"Reinstall `s5cmd` using `pip install s5cmd` in your active Python environment and ensure that the directory where pip installs binaries (e.g., `~/.local/bin` or your virtual environment's `bin` folder) is included in your system's PATH and your shell session is reloaded.","cause":"The `s5cmd` executable is not in your shell's PATH environment variable, even after installing the `s5cmd` Python package.","error":"s5cmd: command not found"},{"fix":"Review your AWS IAM policies and ensure the user or role associated with your `s5cmd` credentials has the required S3 permissions (e.g., `s3:GetObject`, `s3:PutObject`, `s3:ListBucket`) for the target resources.","cause":"The AWS credentials configured for `s5cmd` (via environment variables, shared credentials file, or IAM roles) lack the necessary S3 permissions for the requested operation.","error":"ERROR Access Denied."},{"fix":"Ensure the `s5cmd` configuration file exists at the expected path and is correctly formatted in YAML. If using a custom config file, specify its path explicitly using the `--config-file` argument.","cause":"`s5cmd` could not locate or correctly parse its configuration file, typically `~/.s5cmd.yaml`, or a custom path specified by the user.","error":"Error parsing config file: open ~/.s5cmd.yaml: no such file or directory"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.3.3","cli_name":"s5cmd","cli_version":"Incorrect Usage: flag provided but not defined: -version","type":"library","homepage":null,"github":"https://github.com/jcfr/s5cmd-python-distributions","docs":null,"changelog":"https://github.com/jcfr/s5cmd-python-distributions/releases","pypi":"https://pypi.org/project/s5cmd/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["aws","devops","database"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-30","next_check":"2026-07-28","install_tag":null}}