{"id":792,"library":"pandas-gbq","title":"pandas-gbq","description":"pandas-gbq is a Python library that provides a convenient interface to connect pandas DataFrames with Google BigQuery. It simplifies reading data from BigQuery into a pandas.DataFrame and writing DataFrames to BigQuery tables. The current version is 0.34.1, released on 2026-03-26, and the library maintains a regular release cadence, typically with monthly or bi-monthly updates for new features and bug fixes.","status":"active","version":"0.34.1","language":"python","source_language":"en","source_url":"https://github.com/googleapis/google-cloud-python/tree/main/packages/pandas-gbq","tags":["bigquery","pandas","data-science","google-cloud","etl","data-warehouse"],"install":[{"cmd":"pip install pandas-gbq","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Core DataFrame manipulation library.","package":"pandas"},{"reason":"Google Cloud client library for BigQuery API interactions.","package":"google-cloud-bigquery"},{"reason":"Authentication and authorization for Google's APIs.","package":"google-auth"},{"reason":"Helpers for user-based authentication to Google's API.","package":"pydata-google-auth"},{"reason":"Used for efficient data formatting and transfer, especially with the BigQuery Storage API.","package":"pyarrow","optional":true},{"reason":"Client library for the BigQuery Storage API, enabling faster large data downloads.","package":"google-cloud-bigquery-storage","optional":true},{"reason":"Provides progress bars for data uploads/downloads.","package":"tqdm","optional":true}],"imports":[{"wrong":"import pandas_gbq","symbol":"read_gbq","correct":"from pandas_gbq import read_gbq"},{"wrong":"import pandas_gbq","symbol":"Context","correct":"from pandas_gbq import Context"},{"wrong":"import pandas_gbq","symbol":"query","correct":"from pandas_gbq import query"}],"quickstart":{"code":"import os\nimport pandas as pd\nimport pandas_gbq\n\n# Set your Google Cloud Project ID\n# It's recommended to set this as an environment variable or via credentials\nproject_id = os.environ.get('GOOGLE_CLOUD_PROJECT', 'your-gcp-project-id')\n\n# --- Reading data from BigQuery ---\n# Example query from a public dataset\nsql_query = \"\"\"\n    SELECT country_name, alpha_2_code\n    FROM `bigquery-public-data.utility_us.country_code_iso`\n    WHERE alpha_2_code LIKE 'U%'\n    LIMIT 5\n\"\"\"\n\ntry:\n    df_read = pandas_gbq.read_gbq(sql_query, project_id=project_id)\n    print(\"\\n--- Data read from BigQuery ---\")\n    print(df_read)\nexcept Exception as e:\n    print(f\"Error reading from BigQuery: {e}\")\n    print(\"Please ensure GOOGLE_CLOUD_PROJECT is set and you have authenticated (e.g., `gcloud auth application-default login`).\")\n\n# --- Writing data to BigQuery ---\n# Create a sample DataFrame to upload\ndata = {\n    'col1': [1, 2, 3],\n    'col2': ['A', 'B', 'C'],\n    'timestamp_col': pd.to_datetime(['2026-01-01', '2026-01-02', '2026-01-03'])\n}\ndf_write = pd.DataFrame(data)\n\n# Define destination table (dataset.tablename)\ndestination_table = 'my_test_dataset.my_test_table'\n\n# To avoid errors, you might want to replace the table if it exists for testing\n# In production, consider 'append' or 'fail' with proper checks\ntry:\n    pandas_gbq.to_gbq(\n        df_write,\n        destination_table,\n        project_id=project_id,\n        if_exists='replace' # Options: 'fail', 'replace', 'append'\n    )\n    print(f\"\\n--- DataFrame successfully written to {destination_table} in project {project_id} ---\")\nexcept Exception as e:\n    print(f\"Error writing to BigQuery: {e}\")\n    print(\"Ensure 'my_test_dataset' exists in BigQuery or remove 'my_test_dataset.' from 'destination_table' to allow automatic dataset creation if permitted.\")\n","lang":"python","description":"This quickstart demonstrates how to read data from a public BigQuery dataset into a pandas DataFrame and write a pandas DataFrame to a new BigQuery table. It assumes you have a Google Cloud project set up and have authenticated (e.g., using `gcloud auth application-default login`). The `project_id` is retrieved from the `GOOGLE_CLOUD_PROJECT` environment variable for robustness."},"warnings":[{"fix":"Upgrade to Python 3.9+ and ensure all project dependencies are compatible.","message":"Python 2 support was officially dropped as of January 1, 2020. Versions released after this date require Python 3.9 or higher.","severity":"breaking","affected_versions":"0.20.0 and later"},{"fix":"Set the `GOOGLE_CLOUD_PROJECT` environment variable. Authenticate using `gcloud auth application-default login`, provide a service account JSON file via the `credentials` parameter, or set `pandas_gbq.context.credentials` and `pandas_gbq.context.project` explicitly.","message":"Authentication is critical. Without proper credentials or a `project_id`, `pandas-gbq` will raise errors (e.g., `ValueError: Could not determine project ID`). Common authentication methods include Application Default Credentials (ADC), service account keys, or user-based OAuth.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Review and update BigQuery table schemas if necessary. For `datetime` columns, consider making them timezone-aware (`pd.to_datetime(..., utc=True)`) if `TIMESTAMP` is desired, or explicitly define `table_schema` in `to_gbq`.","message":"The `to_gbq` function has breaking changes in how it infers BigQuery data types for certain pandas dtypes. Naive (timezone-unaware) datetime columns are now loaded as BigQuery `DATETIME` instead of `TIMESTAMP`. Object columns containing boolean or dictionary values are loaded as `BOOLEAN` or `STRUCT` respectively, instead of `STRING`. `UInt8` columns are now `INT64`.","severity":"breaking","affected_versions":"0.34.0 and later"},{"fix":"Explicitly set `if_exists='replace'` to overwrite the table, or `if_exists='append'` to add data to an existing table. Always handle this parameter carefully to prevent unintended data loss or duplication.","message":"When using `to_gbq`, the default `if_exists` parameter is 'fail', meaning the operation will fail if the destination table already exists.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure your environment allows for the local webserver flow (e.g., a browser can open `localhost:808X`). If working in a headless environment, consider using service account authentication.","message":"The `auth_local_webserver` parameter's default behavior changed from `False` to `True` in `pandas-gbq` version 1.5.0. This is due to Google deprecating the 'out-of-band' (copy-paste) authentication flow.","severity":"deprecated","affected_versions":"1.5.0 and later"}],"env_vars":null,"search_vec":"'-03':46 '-26':47 '0.34.1':42 '2026':45 'bi':61 'bi-month':60 'bigqueri':22,28,36,70 'bug':68 'cadenc':55 'cloud':77 'connect':17 'conveni':14 'current':39 'data':26,73,80 'data-sci':72 'data-warehous':79 'datafram':19,34 'etl':78 'featur':66 'fix':69 'gbq':3,6 'googl':21,76 'google-cloud':75 'interfac':15 'librari':10,50 'maintain':51 'month':58,62 'new':65 'panda':2,5,18,71 'pandas-gbq':1,4 'pandas.dataframe':31 'provid':12 'python':9 'read':25 'regular':53 'releas':43,54 'scienc':74 'simplifi':24 'tabl':37 'typic':56 'updat':63 'version':40 'warehous':81 'write':33","created_at":"2026-03-29T04:21:53.995939+00:00","updated_at":"2026-04-16T17:51:57.182110+00:00","problems":null,"ecosystem":"pypi","meta_description":null,"install_score":95,"quickstart_score":60,"quickstart_tag":"reviewed","pypi_latest":"0.35.0","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/googleapis/google-cloud-python","docs":null,"changelog":null,"pypi":"https://pypi.org/project/pandas-gbq/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["gcp","data","database"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-30","last_verified":"2026-06-30","next_check":"2026-07-30","install_tag":"verified"}}