{"id":7144,"library":"dbldatagen","title":"Databricks Labs - PySpark Synthetic Data Generator","description":"dbldatagen (Databricks Labs Data Generator) is an open-source Python library for generating synthetic data at scale within Apache Spark and Databricks environments. It allows users to define complex data schemas with various constraints, distributions, and inter-column relationships to create realistic datasets for testing, benchmarking, and machine learning model development. The library is currently at version 0.4.0.post1 and has an active development and release cadence.","status":"active","version":"0.4.0.post1","language":"python","source_language":"en","source_url":"https://github.com/databrickslabs/data-generator","tags":["pyspark","databricks","synthetic-data","data-generation","testing","benchmarking"],"install":[{"cmd":"pip install dbldatagen","lang":"bash","label":"Standard pip install"},{"cmd":"%pip install dbldatagen","lang":"bash","label":"Databricks Notebook"}],"dependencies":[{"reason":"Core dependency for Spark DataFrame operations, requires >=3.2.1 for dbldatagen v0.4.0.","package":"pyspark","optional":false},{"reason":"May be an implicit dependency in some environments (e.g., Google Colab) to avoid import errors related to JSON processing.","package":"jmespath","optional":true}],"imports":[{"wrong":"import dbldatagen as dg","symbol":"dbldatagen","correct":"import dbldatagen"}],"quickstart":{"code":"from pyspark.sql import SparkSession\nimport dbldatagen as dg\n\n# Initialize SparkSession (if not in Databricks already)\ntry:\n    spark\nexcept NameError:\n    spark = SparkSession.builder.appName(\"dbldatagen_quickstart\").getOrCreate()\n\n# Generate a basic user dataset using standard datasets feature\n# This creates 1 million rows and 4 partitions\nprint(\"Generating a basic user dataset...\")\ndf = dg.Datasets(spark, \"basic/user\").get(rows=1_000_000, partitions=4).build()\n\n# Display schema and a few rows\nprint(\"Schema:\")\ndf.printSchema()\n\nprint(\"Sample data:\")\ndf.show(5, truncate=False)\n\n# Stop SparkSession if it was created here (optional)\n# spark.stop()","lang":"python","description":"This quickstart demonstrates how to generate a synthetic dataset using `dbldatagen`'s `Datasets` class, which provides pre-configured data generation recipes. It initializes a SparkSession (if not already present), creates a 'basic/user' dataset with 1 million rows and 4 partitions, then displays its schema and the first few rows. This approach is recommended for quickly generating common synthetic data patterns."},"warnings":[{"fix":"Upgrade your PySpark installation (`pip install 'pyspark>=3.2.1'`) and ensure your Databricks Runtime is 10.4 LTS or newer.","message":"Version 0.4.0 increased the minimum `pyspark` version to 3.2.1 and requires Databricks runtime 10.4 LTS or later. Older PySpark versions or Databricks runtimes will not be compatible.","severity":"breaking","affected_versions":"0.4.0+"},{"fix":"Ensure consistent casing for column names throughout your data generation specifications to avoid conflicts.","message":"Spark SQL column names are case-insensitive. Defining new columns with the same name but different casing than existing ones may lead to unexpected behavior or errors in downstream operations.","severity":"gotcha","affected_versions":"All"},{"fix":"For stateful deduplication across an entire stream, apply watermarking and deduplication using Spark's native streaming APIs on the DataFrame produced by `build()`.","message":"When using `dbldatagen.constraints.UniqueCombinations` with streaming dataframes, deduplication is performed only within a batch. For full stream-wide deduplication, you must implement explicit watermarking and deduplication logic on the resultant DataFrame, which can be resource-intensive for high-volume streams.","severity":"gotcha","affected_versions":"All"},{"fix":"Customize the internal seed column name by setting the `seedColumnName` attribute when creating the `DataGenerator` instance (e.g., `DataGenerator(..., seedColumnName=\"_internal_id\")`).","message":"The column name 'id' is reserved internally by `dbldatagen` as the seed column for data generation. If your generated data requires a column named 'id' with different semantics, it will conflict with this internal mechanism.","severity":"gotcha","affected_versions":"All"},{"fix":"Use Databricks Runtime 13.2 or later, or configure your cluster to use 'Single User' or 'No Isolation Shared' access modes if using older runtimes.","message":"When running on Databricks Unity Catalog enabled environments with Runtimes prior to 13.2, `dbldatagen` requires 'Single User' or 'No Isolation Shared' access modes. 'Shared' access mode in these older runtimes lacks necessary features (e.g., 3rd party libraries, Python UDFs) for `dbldatagen` to function correctly. This limitation is resolved in Databricks Runtimes 13.2 and newer.","severity":"gotcha","affected_versions":"<13.2 Databricks Runtimes on Unity Catalog"}],"env_vars":null,"search_vec":"'0.4.0':66 'activ':71 'allow':32 'apach':26 'benchmark':54,85 'cadenc':75 'column':46 'complex':36 'constraint':41 'creat':49 'current':63 'data':5,10,22,37,80,82 'data-gener':81 'databrick':1,8,29,77 'dataset':51 'dbldatagen':7 'defin':35 'develop':59,72 'distribut':42 'environ':30 'generat':6,11,20,83 'inter':45 'inter-column':44 'lab':2,9 'learn':57 'librari':18,61 'machin':56 'model':58 'open':15 'open-sourc':14 'post1':67 'pyspark':3,76 'python':17 'realist':50 'relationship':47 'releas':74 'scale':24 'schema':38 'sourc':16 'spark':27 'synthet':4,21,79 'synthetic-data':78 'test':53,84 'user':33 'various':40 'version':65 'within':25","created_at":"2026-04-16T13:45:22.276792+00:00","updated_at":"2026-04-16T13:45:22.276792+00:00","problems":{"verify_error":"Traceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"/tmp/tmp32dgbv3l/venv/lib/python3.12/site-packages/dbldatagen/__init__.py\", line 26, in <module>\n    from .data_generator import DataGenerator\n  File \"/tmp/tmp32dgbv3l/venv/lib/python3.12/site-packages/dbldatagen/data_"},"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.4.0.post1","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/databrickslabs/data-generator","docs":"https://databrickslabs.github.io/dbldatagen/public_docs/index.html","changelog":null,"pypi":"https://pypi.org/project/dbldatagen/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","database","ai-ml","devops"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"import_fail","verified_at":"2026-07-03","last_verified":"2026-07-03","next_check":"2026-07-10","install_tag":null}}