{"id":1217,"library":"apache-airflow-providers-amazon","title":"Apache Airflow Amazon Provider","description":"The `apache-airflow-providers-amazon` package extends Apache Airflow by providing operators, hooks, and sensors for seamless integration with various Amazon Web Services (AWS). It enables orchestrating workflows involving services like Amazon S3, Redshift, EMR, SQS, and more. The provider is actively maintained by the Apache Airflow community with frequent releases, typically every few weeks.","status":"active","version":"9.24.0","language":"python","source_language":"en","source_url":"https://github.com/apache/airflow/tree/main/airflow/providers/amazon","tags":["airflow","aws","amazon","provider","cloud","orchestration","etl"],"install":[{"cmd":"pip install apache-airflow-providers-amazon","lang":"bash","label":"Install core provider"},{"cmd":"pip install 'apache-airflow-providers-amazon[s3fs,cncf.kubernetes]' # Example with optional extras","lang":"bash","label":"Install with optional extras"}],"dependencies":[{"reason":"Core Apache Airflow library; required for provider functionality.","package":"apache-airflow","optional":false},{"reason":"AWS SDK for Python; typically a transitive dependency but essential for AWS interactions.","package":"boto3","optional":false},{"reason":"Required by some operators (e.g., SQL to S3) for data manipulation, can be installed via `[pandas]` extra.","package":"pandas","optional":true}],"imports":[{"symbol":"S3Hook","correct":"from airflow.providers.amazon.aws.hooks.s3 import S3Hook"},{"symbol":"S3CopyObjectOperator","correct":"from airflow.providers.amazon.aws.operators.s3 import S3CopyObjectOperator"},{"symbol":"S3KeySensor","correct":"from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor"},{"note":"The `ecs_operator` module was refactored, and components moved to `ecs` module in some versions.","wrong":"from airflow.providers.amazon.aws.operators.ecs_operator import EcsOperator","symbol":"EcsOperator","correct":"from airflow.providers.amazon.aws.operators.ecs import EcsOperator"}],"quickstart":{"code":"from __future__ import annotations\n\nimport os\n\nimport pendulum\n\nfrom airflow.models.dag import DAG\nfrom airflow.providers.amazon.aws.operators.s3 import S3CopyObjectOperator\n\n# Ensure AWS credentials are configured in Airflow Connection 'aws_default'\n# or via environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION)\n# For local testing, you can mock this or use dummy values if S3 interaction isn't critical.\n\nwith DAG(\n    dag_id=\"example_s3_copy_dag\",\n    start_date=pendulum.datetime(2023, 1, 1, tz=\"UTC\"),\n    catchup=False,\n    schedule=None,\n    tags=[\"s3\", \"aws\", \"example\"],\n) as dag:\n    copy_s3_object = S3CopyObjectOperator(\n        task_id=\"copy_s3_object_task\",\n        source_bucket_key=f\"s3://{os.environ.get('SOURCE_BUCKET', 'my-source-bucket')}/source_file.txt\",\n        dest_bucket_key=f\"s3://{os.environ.get('DEST_BUCKET', 'my-dest-bucket')}/dest_file.txt\",\n        aws_conn_id=\"aws_default\", # Assumes 'aws_default' connection is configured in Airflow\n    )\n","lang":"python","description":"This example demonstrates a basic Airflow DAG that uses the `S3CopyObjectOperator` from the Amazon provider to copy an object between two S3 buckets. Ensure you have an Airflow connection named `aws_default` configured with appropriate AWS credentials and permissions, or set AWS environment variables."},"warnings":[{"fix":"Upgrade your Apache Airflow environment to at least version 2.11.0. Consult the Airflow documentation for upgrade paths.","message":"Provider versions 9.x and above (including 9.24.0) require Apache Airflow version 2.11.0 or greater. Attempting to install on older Airflow versions will lead to compatibility issues or automatic Airflow upgrades.","severity":"breaking","affected_versions":"9.0.0+"},{"fix":"Update existing DAGs to use `cloudformation_parameters` instead of `params` for these operators.","message":"For provider versions 3.0.0 and above, the `CloudFormationCreateStackOperator` and `CloudFormationDeleteStackOperator` had their `params` argument renamed to `cloudformation_parameters`. This was to avoid a clash with Airflow 2.2+'s DAG `params` field.","severity":"breaking","affected_versions":"3.0.0+"},{"fix":"Use the `impersonation_chain` parameter for impersonation in affected operators.","message":"In provider version 9.24.0, the deprecated `delegate_to` parameter has been removed from `GCSToS3Operator`, `GlacierToGCSOperator`, and `GoogleApiToS3Operator`.","severity":"breaking","affected_versions":"9.24.0"},{"fix":"Refer to the latest Airflow AWS provider documentation for recommended connection configurations, preferring `endpoint_url` in `extra` where applicable.","message":"The use of `aws_access_key_id` and `aws_secret_access_key` directly in AWS connection `extra` fields is being deprecated in favor of `endpoint_url` for some connections.","severity":"deprecated","affected_versions":"4.1.0+"},{"fix":"Always explicitly include a `--constraint` statement in your `requirements.txt` for MWAA environments.","message":"When running on Amazon MWAA with Airflow v2.7.2+, your `requirements.txt` file should include a `--constraint` statement. If omitted, MWAA will apply a default constraint file, which might not align with desired provider versions.","severity":"gotcha","affected_versions":"Airflow 2.7.2+"},{"fix":"It is recommended to use a Python version officially supported by the library, typically Python 3.8-3.12, until full Python 3.13 compatibility is confirmed.","message":"Installation or execution of this library on Python 3.13 environments (e.g., python:3.13-alpine) may result in timeouts due to potential compatibility issues with the new Python version.","severity":"breaking","affected_versions":"Python 3.13"}],"env_vars":null,"search_vec":"'activ':47 'airflow':2,8,14,52,61 'amazon':3,10,26,37,63 'apach':1,7,13,51 'apache-airflow-providers-amazon':6 'aw':29,62 'cloud':65 'communiti':53 'emr':40 'enabl':31 'etl':67 'everi':58 'extend':12 'frequent':55 'hook':18 'integr':23 'involv':34 'like':36 'maintain':48 'oper':17 'orchestr':32,66 'packag':11 'provid':4,9,16,45,64 'redshift':39 'releas':56 's3':38 'seamless':22 'sensor':20 'servic':28,35 'sqs':41 'typic':57 'various':25 'web':27 'week':60 'workflow':33","created_at":"2026-04-06T16:55:03.975187+00:00","updated_at":"2026-04-15T20:23:39.406734+00:00","problems":[{"fix":"Update the import statement to: 'from airflow.providers.amazon.aws.transfers.s3_to_s3 import S3ToS3Operator'","cause":"The module path has changed in recent versions of the 'apache-airflow-providers-amazon' package.","error":"ModuleNotFoundError: No module named 'airflow.providers.amazon.aws.operators.s3'"},{"fix":"Change the import to: 'from airflow.providers.amazon.aws.hooks.s3 import S3Hook'","cause":"The 'S3Hook' class has been moved to a different module in the 'apache-airflow-providers-amazon' package.","error":"AttributeError: module 'airflow.providers.amazon.aws.hooks.s3' has no attribute 'S3Hook'"},{"fix":"Use the correct import: 'from airflow.providers.amazon.aws.operators.emr_create_job_flow import EmrCreateJobFlowOperator'","cause":"The 'EmrCreateJobFlowOperator' has been relocated in the 'apache-airflow-providers-amazon' package.","error":"ImportError: cannot import name 'EmrCreateJobFlowOperator' from 'airflow.providers.amazon.aws.operators.emr'"},{"fix":"Install the 'apache-airflow-providers-common-sql' package and import using: 'from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator'","cause":"The 'RedshiftSQLOperator' has been moved to the 'common-sql' provider package.","error":"ImportError: cannot import name 'RedshiftSQLOperator' from 'airflow.providers.amazon.aws.operators.redshift'"},{"fix":"Install the 'apache-airflow-providers-common-sql' package and import using: 'from airflow.providers.common.sql.transfers.s3_to_sql import S3ToSqlOperator'","cause":"The 'S3ToRedshiftOperator' has been moved to the 'common-sql' provider package.","error":"ImportError: cannot import name 'S3ToRedshiftOperator' from 'airflow.providers.amazon.aws.transfers.s3_to_redshift'"}],"ecosystem":"pypi","meta_description":null,"install_score":0,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"9.35.0","cli_name":"airflow","cli_version":"","type":"library","homepage":null,"github":"https://github.com/apache/airflow","docs":"https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.28.0","changelog":"https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.28.0/changelog.html","pypi":"https://pypi.org/project/apache-airflow-providers-amazon/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["workflow","aws","devops","data"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-27","next_check":"2026-07-28","install_tag":"stale"}}