{"id":8961,"library":"dlt-meta","title":"DLT-META Framework","description":"DLT-META is a metadata-driven framework for Databricks Lakeflow Declarative Pipelines, designed to automate the creation and management of bronze and silver data pipelines. It leverages metadata defined in JSON or YAML files to dynamically generate pipeline code, streamlining data engineering workflows. The library is currently at version 0.0.10 and has active, though irregular, release cycles with consistent updates.","status":"active","version":"0.0.10","language":"python","source_language":"en","source_url":"https://github.com/databrickslabs/dlt-meta","tags":["databricks","delta live tables","dlt","metadata-driven","etl","data pipelines","automation","lakeflow"],"install":[{"cmd":"pip install dlt-meta","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"Required runtime environment.","package":"python","optional":false},{"reason":"Required for CLI interactions and deployment to Databricks workspace (v0.213 or later).","package":"databricks-cli","optional":false},{"reason":"Used for parsing YAML metadata files.","package":"PyYAML","optional":true},{"reason":"Core dependency for Python package management.","package":"setuptools","optional":true},{"reason":"Used for interacting with Databricks APIs.","package":"databricks-sdk","optional":true}],"imports":[{"wrong":"from dlt_meta.src import DataflowPipeline","symbol":"DataflowPipeline","correct":"from dlt_meta.src import DataflowPipeline"}],"quickstart":{"code":"# This code typically runs within a Databricks Notebook or job after metadata onboarding.\n# Ensure 'dlt-meta' is installed via %pip install dlt-meta in the notebook or as a cluster library.\n\nimport dlt\nfrom dlt_meta import DataflowPipeline\nimport os\n\n# These parameters would typically be passed as job parameters in Databricks\n# For local testing, you might set environment variables or hardcode.\nlayer = os.environ.get('DLT_META_LAYER', 'bronze').lower() # e.g., 'bronze' or 'silver'\nenv = os.environ.get('DLT_META_ENV', 'dev').lower() # e.g., 'dev', 'qa', 'prod'\n\n# In a Databricks environment, 'spark' session is implicitly available.\n# For local testing outside Databricks, you would need to initialize a SparkSession.\n# Example placeholder for local SparkSession (not typically done in DLT-META's primary use-case):\n# from pyspark.sql import SparkSession\n# spark = SparkSession.builder.appName(\"dlt-meta-local\").getOrCreate()\n\ntry:\n    print(f\"Attempting to invoke DLT-META for layer: {layer} (env: {env}).\")\n    # The 'spark' object is expected to be the Databricks SparkSession\n    DataflowPipeline.invoke_dlt_pipeline(spark=spark, layer=layer, env=env)\n    print(f\"DLT-META successfully invoked for layer: {layer} (env: {env}).\")\nexcept ImportError:\n    print(\"ERROR: Could not import DataflowPipeline from dlt_meta. Ensure the 'dlt-meta' library is installed and available.\")\n    raise\nexcept Exception as e:\n    print(f\"ERROR: An exception occurred during DLT-META pipeline invocation for layer '{layer}' in env '{env}': {e}\")\n    raise\n","lang":"python","description":"This quickstart demonstrates how to programmatically invoke the `dlt-meta` framework within a Databricks environment (typically a notebook or job). It assumes `dlt-meta` is installed and metadata has been onboarded. The `DataflowPipeline.invoke_dlt_pipeline` method orchestrates the creation and execution of DLT pipelines based on the provided layer and environment, reading from pre-configured metadata."},"warnings":[{"fix":"Migrate existing pipelines from DPM mode to the default publishing mode as per Databricks' migration guide.","message":"The DPM (Direct Publishing Mode) flag was removed in v0.0.10. Pipelines using DPM mode in v0.0.9 must be migrated to the default publishing mode before upgrading. This change is metadata-only but irreversible.","severity":"breaking","affected_versions":">=0.0.10"},{"fix":"Remove database qualifiers from table names in your metadata (e.g., change `database.schema.table` to `schema.table` or `table`).","message":"Multi-Level Namespace Changes in v0.0.10. Custom schema qualification in table names is no longer supported; tables must be created without database qualifiers.","severity":"breaking","affected_versions":">=0.0.10"},{"fix":"Update existing pipeline configurations to use the new layer-specific argument prefixes (e.g., `bronze_arg` or `silver_arg`).","message":"Argument changes for `invoke_dlt_pipeline` in v0.0.10. Method arguments now require `bronze_` or `silver_` prefixes to support `apply_changes_from_snapshot` in both layers.","severity":"breaking","affected_versions":">=0.0.10"},{"fix":"For issues, report them directly on the dlt-meta GitHub repository.","message":"DLT-META is a Databricks Labs project and is provided for exploration only. Databricks does not formally support it or provide SLAs. Do not submit Databricks support tickets for issues; instead, file a GitHub issue.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Validate onboarding JSON/YAML against a predefined schema before ingestion. Ensure all required fields are present and data types are correct.","message":"Malformed JSON/YAML metadata can lead to job failures. The framework relies heavily on correct metadata structure and content.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"search_vec":"'0.0.10':56 'activ':59 'autom':21,78 'bronz':27 'code':45 'consist':65 'creation':23 'current':53 'cycl':63 'data':30,47,76 'databrick':15,67 'declar':17 'defin':35 'delta':68 'design':19 'dlt':2,6,71 'dlt-meta':1,5 'driven':12,74 'dynam':42 'engin':48 'etl':75 'file':40 'framework':4,13 'generat':43 'irregular':61 'json':37 'lakeflow':16,79 'leverag':33 'librari':51 'live':69 'manag':25 'meta':3,7 'metadata':11,34,73 'metadata-driven':10,72 'pipelin':18,31,44,77 'releas':62 'silver':29 'streamlin':46 'tabl':70 'though':60 'updat':66 'version':55 'workflow':49 'yaml':39","created_at":"2026-04-16T18:47:44.920895+00:00","updated_at":"2026-04-16T18:47:44.920895+00:00","problems":{"verify_error":"Traceback (most recent call last):\n  File \"<string>\", line 1, in <module>\nModuleNotFoundError: No module named 'dlt_meta'"},"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.0.10","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":null,"docs":null,"changelog":null,"pypi":"https://pypi.org/project/dlt-meta/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","workflow","devops","aws"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"import_fail","verified_at":"2026-07-03","last_verified":"2026-07-03","next_check":"2026-07-10","install_tag":null}}