{"id":10240,"library":"soda-core-bigquery","title":"Soda Core BigQuery","description":"Soda Core BigQuery is an extension for Soda Core, an open-source data quality testing tool. It enables users to define, execute, and monitor data quality checks directly against data stored in Google BigQuery. This package provides the necessary connector and SQL dialect definitions to interact with BigQuery, allowing for comprehensive data quality assessments within a Python environment, typically managed through the Soda CLI and YAML configuration files. The current version is 3.5.6, and it follows the release cadence of the broader Soda Core project.","status":"active","version":"3.5.6","language":"python","source_language":"en","source_url":"https://github.com/sodadata/soda-core","tags":["data quality","bigquery","data governance","testing","analytics","sql","google cloud"],"install":[{"cmd":"pip install soda-core-bigquery","lang":"bash","label":"Install `soda-core-bigquery`"}],"dependencies":[{"reason":"Provides the core Soda CLI, scan execution engine, and base API functionalities; `soda-core-bigquery` is an extension.","package":"soda-core"}],"imports":[{"wrong":"from soda.scan import Scan","symbol":"Scan","correct":"from soda.scan import Scan"}],"quickstart":{"code":"import os\nfrom soda.scan import Scan\n\n# Configure your GCP project ID. For local execution, ensure:\n# 1. 'GOOGLE_APPLICATION_CREDENTIALS' env var points to a service account key JSON,\n#    OR 2. 'gcloud auth application-default login' has been run.\n# os.environ['BIGQUERY_PROJECT_ID'] = 'your-gcp-project-id'\n\n# Initialize a Soda Scan\nscan = Scan()\n\n# Add BigQuery data source configuration via a YAML string\n# Replace 'my-dummy-project' with your actual GCP project ID\nscan.add_configuration_yaml_str(f'''\ndata_sources:\n  bigquery_source:\n    type: bigquery\n    project_id: {os.environ.get('BIGQUERY_PROJECT_ID', 'my-dummy-project')} \n''')\n\n# Define data quality checks via a YAML string\n# Replace 'my_dataset.my_table' with an actual BigQuery dataset.table for real checks\nscan.add_checks_yaml_str('''\nchecks for my_dataset.my_table:\n  - row_count > 0: # Checks if the table has any rows\n      name: 'Table should not be empty'\n  - missing_count(id) = 0: # Checks for missing values in 'id' column\n      name: 'No missing IDs'\n  - duplicate_count(id) = 0: # Checks for duplicate values in 'id' column\n      name: 'No duplicate IDs'\n''')\n\nprint(\"Running Soda Scan...\")\nscan.execute()\n\nif scan.has_failures():\n    print(\"\\nSoda Scan finished with failures.\")\nelse:\n    print(\"\\nSoda Scan finished with no failures.\")\n\n# You can inspect the scan results for detailed outcomes\n# print(scan.get_scan_results())","lang":"python","description":"This quickstart demonstrates how to run a Soda Scan programmatically against a BigQuery data source. It configures the BigQuery connection and defines simple data quality checks. For this to run successfully against your data, ensure BigQuery authentication is set up (e.g., `GOOGLE_APPLICATION_CREDENTIALS` environment variable or `gcloud auth application-default login`) and replace placeholder values for `project_id`, `dataset`, and `table` with your actual BigQuery resources."},"warnings":[{"fix":"Refer to the official Soda Core 3.x migration guide for CLI and YAML syntax updates. Always test configurations in a development environment before deploying to production.","message":"Soda Core 3.x introduced significant changes to the CLI commands (e.g., `soda scan` replaced `soda analyze`) and the structure of configuration YAML files compared to 2.x versions. Migrating from older versions requires updating commands and YAML definitions.","severity":"breaking","affected_versions":"2.x to 3.x"},{"fix":"Ensure the service account or user account has at least `BigQuery Data Viewer` (to read data) and `BigQuery Job User` (to run queries) roles. For service accounts, verify `GOOGLE_APPLICATION_CREDENTIALS` points to a valid key JSON file. For user credentials, ensure `gcloud auth application-default login` has been executed.","message":"Proper BigQuery authentication and IAM roles are critical. Common issues include incorrect service account keys, missing `GOOGLE_APPLICATION_CREDENTIALS` environment variable, or insufficient IAM permissions for the service account/user.","severity":"gotcha","affected_versions":"All"},{"fix":"If encountering issues, explicitly check `pip show soda-core` and `pip show soda-core-bigquery` to confirm both are installed and compatible. Rarely, direct `pip install soda-core` might be needed if `soda-core-bigquery`'s dependency resolution fails.","message":"`soda-core-bigquery` is an extension package. While `pip install soda-core-bigquery` typically pulls `soda-core` as a dependency, the core functionalities (CLI, `Scan` object) are provided by `soda-core`. Ensure `soda-core` is available and compatible.","severity":"gotcha","affected_versions":"All"},{"fix":"Use a YAML linter or validator. Carefully review the Soda Core documentation for the exact YAML structure and examples. Pay close attention to spacing and nesting, especially for data source definitions and check configurations.","message":"YAML configuration files (`configuration.yml`, `checks.yml`) are highly sensitive to indentation and syntax. Minor errors can lead to failures, unexecuted checks, or incorrect interpretations without clear error messages.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"search_vec":"'3.5.6':77 'allow':53 'analyt':96 'assess':58 'bigqueri':3,6,38,52,92 'broader':86 'cadenc':83 'check':31 'cli':68 'cloud':99 'comprehens':55 'configur':71 'connector':44 'core':2,5,12,88 'current':74 'data':17,29,34,56,90,93 'defin':25 'definit':48 'dialect':47 'direct':32 'enabl':22 'environ':62 'execut':26 'extens':9 'file':72 'follow':80 'googl':37,98 'govern':94 'interact':50 'manag':64 'monitor':28 'necessari':43 'open':15 'open-sourc':14 'packag':40 'project':89 'provid':41 'python':61 'qualiti':18,30,57,91 'releas':82 'soda':1,4,11,67,87 'sourc':16 'sql':46,97 'store':35 'test':19,95 'tool':20 'typic':63 'user':23 'version':75 'within':59 'yaml':70","created_at":"2026-04-17T01:23:05.605414+00:00","updated_at":"2026-04-17T01:23:05.605414+00:00","problems":{"verify_error":"Traceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"/tmp/tmp62n0cqef/venv/lib/python3.12/site-packages/soda/scan.py\", line 12, in <module>\n    from soda.common.env_helper import EnvHelper\n  File \"/tmp/tmp62n0cqef/venv/lib/python3.12/site-packages/soda/common/env_helper."},"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"3.5.6","cli_name":"soda","cli_version":"soda-core, version 3.5.6","type":"library","homepage":"https://www.soda.io","github":null,"docs":null,"changelog":null,"pypi":"https://pypi.org/project/soda-core-bigquery/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["testing","gcp","data"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"import_fail","verified_at":"2026-07-03","last_verified":"2026-07-03","next_check":"2026-07-10","install_tag":null}}