{"id":7146,"library":"dbt-glue","title":"dbt-glue adapter for AWS Glue","description":"dbt-glue is a dbt adapter that enables data analysts and engineers to transform data using AWS Glue's Spark engine and interactive sessions. It supports various file formats, including Apache Iceberg, Delta Lake, and Apache Hudi, allowing users to build and manage data pipelines in an AWS data lake environment. The library is actively maintained with frequent updates, aligning with `dbt-core` releases. It is currently at version 1.10.19 and requires Python >=3.9.","status":"active","version":"1.10.19","language":"python","source_language":"en","source_url":"https://github.com/aws-samples/dbt-glue","tags":["dbt","AWS Glue","data transformation","ETL","data lake","Spark","Iceberg","Delta Lake","Hudi"],"install":[{"cmd":"pip install dbt-core dbt-glue","lang":"bash","label":"Install dbt-glue with dbt-core"}],"dependencies":[{"reason":"dbt-glue is an adapter for dbt Core, which provides the main CLI and framework.","package":"dbt-core","optional":false},{"reason":"dbt-glue leverages dbt-spark for its underlying Spark compatibility, especially in earlier versions.","package":"dbt-spark","optional":true}],"imports":[{"note":"Used within Python models for model configuration.","symbol":"dbt.config","correct":"def model(dbt, spark):\n    dbt.config(materialized='python_model', file_format='iceberg')"},{"note":"Used in dbt SQL and Python models to reference other dbt models.","symbol":"dbt.ref","correct":"source_df = dbt.ref(\"my_sql_model\")"},{"note":"Used in dbt SQL and Python models to reference declared data sources.","symbol":"dbt.source","correct":"raw_data = dbt.source('my_source', 'my_table')"}],"quickstart":{"code":"import os\n\n# profiles.yml example for dbt-glue\nprofiles_yml_content = \"\"\"\ndbt_glue_project:\n  target: dev\n  outputs:\n    dev:\n      type: glue\n      query-comment: dbt-glue-example\n      role_arn: \"{{ env_var('DBT_ROLE_ARN', 'arn:aws:iam::123456789012:role/GlueInteractiveSessionRole') }}\"\n      region: \"{{ env_var('AWS_REGION', 'us-east-1') }}\"\n      workers: 5\n      worker_type: G.1X\n      schema: dbt_glue_demo_schema\n      database: dbt_glue_demo_db\n      session_provisioning_timeout_in_seconds: 120\n      location: \"{{ env_var('DBT_S3_LOCATION', 's3://your-s3-bucket/dbt-glue/') }}\"\n      glue_version: \"4.0\"\n      conf: \"--conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO --conf spark.sql.catalog.glue_catalog.lock-impl=org.apache.iceberg.aws.glue.DynamoLockManager --conf spark.sql.catalog.glue_catalog.lock.table=DbtGlueLockTable --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions\"\n\"\"\"\n\n# Example dbt_project.yml (assuming project name 'dbt_glue_project')\ndbt_project_yml_content = \"\"\"\nname: 'dbt_glue_project'\nversion: '1.0.0'\nconfig-version: 2\n\nprofile: 'dbt_glue_project'\n\nmodel-paths: [\"models\"]\nanalysis-paths: [\"analyses\"]\ntest-paths: [\"tests\"]\nseed-paths: [\"seeds\"]\nmacro-paths: [\"macros\"]\nsnapshot-paths: [\"snapshots\"]\n\ntarget-path: \"target\"  # directory which will store compiled SQL files\nclean-targets:\n  - \"target\"\n  - \"dbt_packages\"\n\nmodels:\n  dbt_glue_project:\n    +materialized: table\n\"\"\"\n\n# Example SQL model (models/my_first_model.sql)\nsql_model_content = \"\"\"\n{{ config(materialized='table', file_format='parquet') }}\n\nSELECT\n  1 as id,\n  'dbt-glue' as name\n\"\"\"\n\n# Example Python model (models/my_python_model.py) - requires AWS Glue 4.0+ and Iceberg\npython_model_content = \"\"\"\nfrom pyspark.sql.functions import lit\n\ndef model(dbt, spark):\n    dbt.config(\n        materialized='incremental',\n        file_format='iceberg',\n        unique_key=['id'],\n        incremental_strategy='merge'\n    )\n\n    if dbt.is_incremental():\n        max_id_query = f\"SELECT coalesce(max(id), 0) FROM {dbt.this}\"\n        max_id = spark.sql(max_id_query).collect()[0][0]\n        return spark.createDataFrame([(max_id + 1, 'new_incremental_record')]) \\\n               .toDF(\"id\", \"name\")\n    else:\n        return spark.createDataFrame([(1, 'initial_record'), (2, 'another_initial')]) \\\n               .toDF(\"id\", \"name\")\n\"\"\"\n\n# To run: \n# 1. Ensure AWS credentials and DBT_ROLE_ARN, DBT_S3_LOCATION environment variables are set.\n# 2. Create the project structure: ~/.dbt/profiles.yml, dbt_project.yml, models/my_first_model.sql, models/my_python_model.py\n# 3. dbt debug\n# 4. dbt run","lang":"python","description":"To get started with `dbt-glue`, you'll need to configure your `profiles.yml` to specify connection details for AWS Glue interactive sessions, including the IAM role, region, worker types, and S3 location. SQL models define transformations, and experimental Python models allow for more complex logic using PySpark DataFrames. Ensure `DBT_ROLE_ARN` and `DBT_S3_LOCATION` environment variables are set for authentication and storage paths respectively."},"warnings":[{"fix":"Always install `dbt-core` and `dbt-glue` together: `pip install dbt-core dbt-glue`.","message":"Beginning with dbt Core v1.8, installing a dbt adapter no longer automatically installs `dbt-core`. You must explicitly install both `dbt-core` and `dbt-glue` to avoid missing dependencies or version conflicts.","severity":"breaking","affected_versions":"dbt-core >=1.8.0, dbt-glue >=1.8.0"},{"fix":"Review the official dbt-glue documentation and GitHub README for the latest status and specific requirements before relying on these experimental features in production. Ensure your AWS Glue environment is version 4.0 or higher.","message":"Python model support and Amazon S3 Tables support are currently experimental. They may have limitations or breaking changes in future versions and require AWS Glue 4.0+ for optimal support, and Iceberg file format for Python models.","severity":"gotcha","affected_versions":"All versions with Python/S3 Tables support (from 1.10.9 onwards)"},{"fix":"For issues with table discovery, particularly in tests, ensure Iceberg-specific Spark configurations are correctly set in `profiles.yml` and consider explicitly using the `glue_catalog.` prefix where direct table references are made.","message":"When working with Iceberg tables on AWS Glue, especially in dbt tests or certain queries, you might need to explicitly prefix table names with `glue_catalog.` (e.g., `glue_catalog.your_database.your_table`) in custom SQL or specific configurations if not handled automatically by the adapter's macros.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Update your YAML files to remove duplicate keys and ensure proper Jinja syntax. Use `--select` or `-s` instead of `--models` for CLI commands. If using `--warn-error`, configure `warn-error-options` to handle deprecations appropriately.","message":"dbt Core v1.10 introduces deprecation warnings for several patterns, including duplicate keys in the same YAML file, unexpected Jinja blocks, and the `--models` / `--model` / `-m` CLI flags (which were renamed to `--select` / `--s` in v0.21).","severity":"deprecated","affected_versions":"dbt-core >=1.10.0, dbt-glue >=1.10.0"},{"fix":"Ensure the IAM role associated with your Glue jobs has the necessary permissions for S3 access, Glue Data Catalog operations, and Lake Formation if applicable. A least-privileged policy example is often available in the dbt-glue documentation.","message":"Incorrect IAM permissions for the Glue interactive session role can lead to `AccessDeniedException` errors, preventing dbt-glue from accessing S3 buckets or the Glue Data Catalog.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"search_vec":"'1.10.19':79 '3.9':83 'activ':63 'adapt':4,14 'align':68 'allow':46 'analyst':18 'apach':39,44 'aw':6,25,56,85 'build':49 'core':72 'current':76 'data':17,23,52,57,87,90 'dbt':2,9,13,71,84 'dbt-core':70 'dbt-glue':1,8 'delta':41,94 'enabl':16 'engin':20,29 'environ':59 'etl':89 'file':36 'format':37 'frequent':66 'glue':3,7,10,26,86 'hudi':45,96 'iceberg':40,93 'includ':38 'interact':31 'lake':42,58,91,95 'librari':61 'maintain':64 'manag':51 'pipelin':53 'python':82 'releas':73 'requir':81 'session':32 'spark':28,92 'support':34 'transform':22,88 'updat':67 'use':24 'user':47 'various':35 'version':78","created_at":"2026-04-16T13:45:27.283021+00:00","updated_at":"2026-04-16T13:45:27.283021+00:00","problems":{"verify_error":"no import statement found"},"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"1.10.19","cli_name":"dbt","cli_version":"Core:","type":"library","homepage":null,"github":"https://github.com/aws-samples/dbt-glue","docs":null,"changelog":null,"pypi":"https://pypi.org/project/dbt-glue/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","aws","devops"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"skip","verified_at":"2026-06-28","last_verified":"2026-06-28","next_check":"2026-07-05","install_tag":null}}