{"id":7125,"library":"dagster-gcp-pandas","title":"Dagster GCP Pandas","description":"The `dagster-gcp-pandas` library provides an I/O manager for persisting Pandas DataFrames to Google Cloud Storage (GCS) within Dagster assets. It leverages `pandas` and `gcsfs` for efficient data serialization (defaulting to Parquet). This library is part of the `dagster` ecosystem and its versioning is tightly coupled with the core `dagster` library.","status":"active","version":"0.29.0","language":"python","source_language":"en","source_url":"https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-gcp-pandas","tags":["dagster","gcp","pandas","io-manager","storage","data-engineering","cloud"],"install":[{"cmd":"pip install dagster-gcp-pandas","lang":"bash","label":"Install dagster-gcp-pandas"}],"dependencies":[{"reason":"Core Dagster framework, required for defining assets and resources.","package":"dagster"},{"reason":"Provides base GCS resource functionality which dagster-gcp-pandas builds upon.","package":"dagster-gcp"},{"reason":"The core data structure (DataFrame) that this library manages.","package":"pandas"},{"reason":"Python client for Google Cloud Storage, often required for underlying GCS interactions and authentication.","package":"google-cloud-storage","optional":true}],"imports":[{"wrong":"from dagster_gcp_pandas import GCSPandasIOManager","symbol":"BigQueryPandasIOManager","correct":"from dagster_gcp_pandas import BigQueryPandasIOManager"},{"symbol":"BigQueryPandasTypeHandler","correct":"from dagster_gcp_pandas import BigQueryPandasTypeHandler"},{"symbol":"bigquery_pandas_io_manager","correct":"from dagster_gcp_pandas import bigquery_pandas_io_manager"}],"quickstart":{"code":"import pandas as pd\nfrom dagster import asset, Definitions\nfrom dagster_gcp_pandas import GCSPandasIOManager\nimport os\n\n@asset\ndef my_pandas_dataframe_asset() -> pd.DataFrame:\n    \"\"\"Produces a Pandas DataFrame.\"\"\"\n    return pd.DataFrame({\"value\": [1, 2, 3], \"label\": [\"A\", \"B\", \"C\"]})\n\n# Configure the GCSPandasIOManager to store DataFrames in a specified GCS bucket.\n# Ensure the GCS_BUCKET_NAME environment variable is set or replace \"your-gcs-bucket-name\".\n# You also need appropriate GCP credentials configured (e.g., GOOGLE_APPLICATION_CREDENTIALS).\ngcs_io_manager = GCSPandasIOManager(\n    gcs_bucket=os.environ.get(\"GCS_BUCKET_NAME\", \"your-gcs-bucket-name\"),\n    gcs_prefix=\"dagster_assets/pandas\"\n)\n\ndefs = Definitions(\n    assets=[my_pandas_dataframe_asset],\n    resources={\n        \"io_manager\": gcs_io_manager\n    }\n)\n\n# To run:\n# 1. Save this code as a Python file (e.g., my_project/repo.py)\n# 2. Set the GCS_BUCKET_NAME environment variable:\n#    export GCS_BUCKET_NAME=\"your-actual-bucket-name\"\n# 3. Ensure GCP credentials are set up (e.g., using `gcloud auth application-default login`\n#    or `GOOGLE_APPLICATION_CREDENTIALS` environment variable pointing to a service account key).\n# 4. Execute: `dagster dev -f my_project/repo.py`\n# 5. Navigate to the Dagster UI (usually http://localhost:3000) and materialize the asset.","lang":"python","description":"This quickstart demonstrates how to define an asset that produces a Pandas DataFrame and uses `GCSPandasIOManager` to store it in a Google Cloud Storage bucket. It requires a GCS bucket to be configured and proper GCP authentication."},"warnings":[{"fix":"Always install `dagster-gcp-pandas` and `dagster` with matching major and minor versions (e.g., `pip install dagster==1.x.y dagster-gcp-pandas==0.x.y`). Refer to the Dagster release notes for the correct library version mapping.","message":"Dagster libraries, including `dagster-gcp-pandas`, are versioned in lockstep with the core `dagster` library. Installing mismatched versions (e.g., `dagster==1.0.0` with `dagster-gcp-pandas==0.15.0`) can lead to `ModuleNotFoundError` or other runtime errors.","severity":"breaking","affected_versions":"<1.0.0 (old library versions with new core)"},{"fix":"Ensure the environment where Dagster runs has access to GCP credentials (e.g., `GOOGLE_APPLICATION_CREDENTIALS` environment variable, default credentials for GCE instances, or `gcloud auth application-default login`). The service account/user needs `Storage Object Viewer` and `Storage Object Creator` (or `Storage Object Admin`) roles on the target GCS bucket.","message":"Proper Google Cloud Platform (GCP) authentication and permissions are required for `dagster-gcp-pandas` to interact with GCS. Lack of credentials or insufficient permissions will result in `PermissionDenied` errors.","severity":"gotcha","affected_versions":"All"},{"fix":"When initializing `GCSPandasIOManager`, set `file_extension` explicitly, e.g., `GCSPandasIOManager(gcs_bucket=\"my-bucket\", file_extension=\".csv\")`.","message":"The `GCSPandasIOManager` defaults to Parquet format for serialization. If you expect or require other formats like CSV, JSON, or feather, you must explicitly configure the `file_extension` parameter.","severity":"gotcha","affected_versions":"All"},{"fix":"Carefully double-check the `gcs_bucket` and `gcs_prefix` values in your `GCSPandasIOManager` configuration. Ensure the bucket name is correct and the prefix matches where the data is expected to be stored/retrieved.","message":"Misconfiguring the `gcs_bucket` or `gcs_prefix` parameters can lead to assets not being found when loading, or being written to unexpected locations within GCS.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"search_vec":"'asset':25 'cloud':20,67 'core':54 'coupl':51 'dagster':1,6,24,44,55,57 'dagster-gcp-panda':5 'data':33,65 'data-engin':64 'datafram':17 'default':35 'ecosystem':45 'effici':32 'engin':66 'gcp':2,7,58 'gcs':22 'gcsfs':30 'googl':19 'i/o':12 'io':61 'io-manag':60 'leverag':27 'librari':9,39,56 'manag':13,62 'panda':3,8,16,28,59 'parquet':37 'part':41 'persist':15 'provid':10 'serial':34 'storag':21,63 'tight':50 'version':48 'within':23","created_at":"2026-04-16T13:44:33.126269+00:00","updated_at":"2026-04-16T13:44:33.126269+00:00","problems":null,"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.29.9","cli_name":"","cli_version":null,"type":"library","homepage":"https://dagster.io","github":"https://github.com/dagster-io/dagster","docs":null,"changelog":null,"pypi":"https://pypi.org/project/dagster-gcp-pandas/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","gcp","workflow"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-30","last_verified":"2026-06-30","next_check":"2026-07-30","install_tag":null}}