{"id":1614,"library":"pandera","title":"Pandera Data Validation","description":"Pandera is a lightweight and flexible open-source Python library for data validation and testing statistical data objects, such as Pandas DataFrames and Series. It allows users to define schema objects to validate the structure, types, and values of data, ensuring data quality and preventing unexpected errors. The library is actively maintained, with version 0.30.1 currently available, and undergoes frequent minor releases.","status":"active","version":"0.30.1","language":"python","source_language":"en","source_url":"https://github.com/pandera-dev/pandera","tags":["data validation","pandas","polars","pyspark","schema","testing","data quality"],"install":[{"cmd":"pip install pandera","lang":"bash","label":"Core installation"},{"cmd":"pip install pandera[polars]","lang":"bash","label":"With Polars support"},{"cmd":"pip install pandera[pyspark]","lang":"bash","label":"With PySpark support"}],"dependencies":[{"reason":"Core DataFrame validation backend","package":"pandas","optional":false},{"reason":"Used for numerical operations and type handling","package":"numpy","optional":false},{"reason":"Optional backend for Polars DataFrame validation","package":"polars","optional":true},{"reason":"Optional backend for PySpark DataFrame validation","package":"pyspark","optional":true}],"imports":[{"symbol":"pandera","correct":"import pandera as pa"},{"symbol":"DataFrameSchema","correct":"from pandera import DataFrameSchema"},{"symbol":"Column","correct":"from pandera import Column"},{"symbol":"Check","correct":"from pandera import Check"},{"symbol":"SeriesSchema","correct":"from pandera import SeriesSchema"}],"quickstart":{"code":"import pandas as pd\nimport pandera as pa\nfrom pandera import Column, DataFrameSchema, Check\n\n# 1. Define a DataFrameSchema\nschema = DataFrameSchema(\n    columns={\n        \"id\": Column(int, Check.greater_than_or_equal_to(0)),\n        \"name\": Column(str, Check.str_matches(r\"^[A-Za-z]+$\")),\n        \"value\": Column(float, Check.in_range(0.0, 1.0))\n    },\n    # Optionally specify index validation\n    index=pa.Index(int, name=\"index\"),\n    # Ensure no extra columns exist\n    strict=True\n)\n\n# 2. Create a valid DataFrame\nvalid_df = pd.DataFrame({\n    \"id\": [1, 2, 3],\n    \"name\": [\"Alice\", \"Bob\", \"Charlie\"],\n    \"value\": [0.1, 0.5, 0.9]\n})\n\n# 3. Validate the DataFrame\ntry:\n    validated_df = schema.validate(valid_df)\n    print(\"Valid DataFrame validated successfully:\")\n    print(validated_df)\nexcept pa.errors.SchemaErrors as e:\n    print(f\"Validation failed unexpectedly for valid data: {e}\")\n\n# 4. Create an invalid DataFrame to demonstrate error handling\ninvalid_df = pd.DataFrame({\n    \"id\": [-1, 2, 3], # Fails 'greater_than_or_equal_to(0)'\n    \"name\": [\"Alice\", \"Bob1\", \"Charlie\"], # Fails 'str_matches'\n    \"value\": [0.1, 0.5, 1.5], # Fails 'in_range'\n    \"extra_col\": [1, 2, 3] # Fails 'strict=True'\n})\n\ntry:\n    schema.validate(invalid_df)\nexcept pa.errors.SchemaErrors as e:\n    print(\"\\nInvalid DataFrame caught by schema errors:\")\n    print(e.failure_cases)\n    print(f\"Total errors: {e.n_failures}\")\n","lang":"python","description":"This quickstart defines a `DataFrameSchema` with column and index constraints, then demonstrates validating both a valid and an invalid Pandas DataFrame. It also shows how to catch `SchemaErrors` and inspect `failure_cases`."},"warnings":[{"fix":"Upgrade to Python >= 3.10 or pin Pandera to <0.27.0.","message":"Pandera dropped support for Python 3.9 in version 0.27.0. Users on Python 3.9 must use an older Pandera version.","severity":"breaking","affected_versions":"<0.27.0"},{"fix":"Upgrade Pandera to >=0.27.1 if using Numpy >=2.4.0, or pin Numpy to an earlier version.","message":"Pandera v0.27.1 fixed a regression with `numpy==2.4.0`. Users on `pandera==0.27.0` paired with `numpy==2.4.0` may encounter `ValueError` related to type recognition.","severity":"breaking","affected_versions":"0.27.0"},{"fix":"If using Pandas 3.0+, ensure Pandera is updated to version 0.30.0 or later. If sticking to older Pandas, ensure your Pandera version is compatible (e.g., Pandera <0.30.0 with Pandas <3.0).","message":"Pandera v0.30.0 introduced support for Pandas >=3.0. While this is an enhancement, older Pandera versions (<0.30.0) are not compatible with Pandas 3.0 and will likely fail.","severity":"gotcha","affected_versions":"<0.30.0"},{"fix":"Install with appropriate extras, e.g., `pip install pandera[polars]` or `pip install pandera[pyspark]`.","message":"To use Pandera with alternative DataFrame backends like Polars or PySpark, you must install additional dependencies via extras (e.g., `pip install pandera[polars]`). Core installation only supports Pandas.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"search_vec":"'0.30.1':59 'activ':55 'allow':30 'avail':61 'current':60 'data':2,16,21,44,46,67,74 'datafram':26 'defin':33 'ensur':45 'error':51 'flexibl':9 'frequent':64 'librari':14,53 'lightweight':7 'maintain':56 'minor':65 'object':22,35 'open':11 'open-sourc':10 'panda':25,69 'pandera':1,4 'polar':70 'prevent':49 'pyspark':71 'python':13 'qualiti':47,75 'releas':66 'schema':34,72 'seri':28 'sourc':12 'statist':20 'structur':39 'test':19,73 'type':40 'undergo':63 'unexpect':50 'user':31 'valid':3,17,37,68 'valu':42 'version':58","created_at":"2026-04-09T03:55:41.508018+00:00","updated_at":"2026-04-16T17:52:21.131998+00:00","problems":[{"fix":"Examine the error message, specifically the 'failure cases', to identify which values or rows violated the schema. Adjust the data to conform to the schema or modify the schema if the data is intentionally different. Using `lazy=True` in `validate()` will collect all errors into a `SchemaErrors` exception, providing a comprehensive report of all failures instead of stopping at the first one.","cause":"This is the most common error in Pandera, indicating that a DataFrame or Series failed to meet the validation constraints defined in its schema, such as incorrect data types, values out of range, or failing custom checks.","error":"pandera.errors.SchemaError: <Schema Column(name=..., type=DataType(...))> failed element-wise validator ..."},{"fix":"Ensure you are calling `.validate()` on an instance of a Pandera schema or model class. If you're using a dictionary to define your schema, you must pass it to `pa.DataFrameSchema()` first before calling `validate`.","cause":"This error typically occurs when the `validate` method is called on a Python dictionary or another non-Pandera object, instead of a properly instantiated `DataFrameSchema`, `SeriesSchema`, or `DataFrameModel` object.","error":"AttributeError: 'dict' object has no attribute 'validate'"},{"fix":"To allow nulls in an integer column, explicitly set `nullable=True` in the `Column` definition and use a nullable integer type like `pd.Int64Dtype()` (available in pandas 0.24+) or `pandera.Int` from `pandera.typing`. Alternatively, use `float` if decimal values are acceptable.","cause":"This error often arises when an integer column contains or is expected to contain `NaN` (null) values. Pandas' `int64` dtype does not support `NaN`, so such columns are typically coerced to `float64` or `object` by Pandas, leading to a type mismatch with the `pandera` schema.","error":"pandera.errors.SchemaError: expected series 'column_name' to have type 'int64', got 'object'"},{"fix":"Either add the missing column to your DataFrame or mark the column as optional in your schema by using `pa.Column(..., required=False)` or, if using `DataFrameModel`, by annotating it with `typing.Optional`.","cause":"This error indicates that a column specified as `required` in the Pandera schema is missing from the DataFrame being validated. By default, all columns defined in a schema are considered required.","error":"pandera.errors.SchemaError: column 'column_name' not in dataframe"},{"fix":"If the unexpected columns should be allowed, remove `strict=True` from your schema definition. If the unexpected columns should be dropped, use `strict='filter'` in your schema. If they are truly errors, you must remove them from the DataFrame before validation.","cause":"This error occurs when the `strict=True` option is set in `DataFrameSchema` (or `DataFrameModel.Config`), which enforces that the DataFrame must contain *only* the columns explicitly defined in the schema. Any additional columns will trigger this error.","error":"pandera.errors.SchemaError: SeriesSchema: did not expect column(s) ['unexpected_column']"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.31.1","cli_name":"","cli_version":null,"type":"library","homepage":"https://pandera.readthedocs.io","github":"https://github.com/pandera-dev/pandera","docs":"https://pandera.readthedocs.io","changelog":null,"pypi":"https://pypi.org/project/pandera/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","testing","serialization"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-06-28","next_check":"2026-07-28","install_tag":null}}