{"id":2920,"library":"datafusion","title":"Apache DataFusion Python","description":"A Python library that provides bindings to the Apache Arrow in-memory query engine, DataFusion. It enables users to build and execute high-performance queries using SQL or a DataFrame API against various data sources, including CSV, Parquet, JSON, and in-memory data. Leveraging its Rust-written query engine, it focuses on efficient, zero-copy data exchange with PyArrow. The library is actively maintained, with a current version of 52.3.0, and typically releases in sync with the core DataFusion project.","status":"active","version":"52.3.0","language":"python","source_language":"en","source_url":"https://github.com/apache/datafusion-python","tags":["data processing","query engine","SQL","dataframe","apache arrow","rust","etl"],"install":[{"cmd":"pip install datafusion","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"Core data format and interoperability.","package":"pyarrow","optional":false},{"reason":"Commonly used for converting DataFusion results to Pandas DataFrames.","package":"pandas","optional":true},{"reason":"Required for interacting with Delta Lake tables.","package":"deltalake","optional":true},{"reason":"Required for interacting with Iceberg tables.","package":"pyiceberg","optional":true}],"imports":[{"symbol":"SessionContext","correct":"from datafusion import SessionContext"},{"note":"Used for DataFrame API operations, especially column selection and expressions.","symbol":"col","correct":"from datafusion import col"},{"note":"For defining User-Defined Scalar Functions (UDFs).","symbol":"udf","correct":"from datafusion import udf"},{"note":"Provides access to built-in DataFusion functions like `functions.sum()`.","symbol":"functions","correct":"from datafusion import functions"}],"quickstart":{"code":"from datafusion import SessionContext, col\nimport pyarrow as pa\n\n# Create a DataFusion session context\nctx = SessionContext()\n\n# Create an in-memory PyArrow table\ndata = {\n    \"id\": [1, 2, 3, 4],\n    \"value\": [10, 20, 15, 25],\n    \"category\": [\"A\", \"B\", \"A\", \"C\"]\n}\npyarrow_table = pa.table(data)\n\n# Register the PyArrow table as a DataFusion table\nctx.register_record_batches(\"my_table\", [pyarrow_table.to_batches()])\n\n# Execute a SQL query\ndf_sql = ctx.sql(\"SELECT category, SUM(value) FROM my_table GROUP BY category ORDER BY category\")\nprint(\"SQL Query Result:\")\nprint(df_sql.to_pandas())\n\n# Execute a DataFrame API query\ndf_dataframe = ctx.table(\"my_table\")\ndf_dataframe = df_dataframe.group_by(col(\"category\")) \\\n                             .aggregate([col(\"value\").sum().alias(\"total_value\")]) \\\n                             .sort(col(\"category\"))\nprint(\"\\nDataFrame API Query Result:\")\nprint(df_dataframe.to_pandas())\n","lang":"python","description":"Demonstrates how to create an in-memory PyArrow table, register it with DataFusion's `SessionContext`, and then query it using both SQL and the DataFrame API. Results are converted to Pandas DataFrames for easy display."},"warnings":[{"fix":"Update custom FFI implementations to include `LogicalExtensionCodec` and `TaskContextProvider` and adapt to new function signatures. Refer to the DataFusion Python Extensions documentation for migration details.","message":"Breaking changes to Foreign Function Interface (FFI) for Python extensions (e.g., custom CatalogProvider, TableProvider). Users implementing custom FFI-based providers must now provide `LogicalExtensionCodec` and `TaskContextProvider`, and method signatures have changed.","severity":"breaking","affected_versions":">= 52.0.0"},{"fix":"Carefully manage dependencies and their DataFusion version requirements. Consider using `pip freeze` and `pip check` to identify conflicts. Check release notes of downstream libraries for compatible DataFusion versions.","message":"DataFusion's Python bindings are tightly coupled with the core Rust DataFusion library. Downstream libraries (e.g., `deltalake`, `pyiceberg`) that provide DataFusion table providers often require exact version matches. This can lead to dependency conflicts when using multiple such libraries.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Adjust custom `FileSource` and `FileScanConfigBuilder` implementations to provide schemas upfront during construction. Update `FilePruner` usage as per the migration guide.","message":"The way schemas are passed to `FileSource` constructors and `FileScanConfigBuilder` has been refactored. File sources now require the schema (including partition columns) at construction, and `FileScanConfigBuilder` no longer accepts a separate schema parameter. Additionally, `FilePruner::try_new()` signature changed.","severity":"breaking","affected_versions":">= 44.0.0"},{"fix":"Remove reliance on `SchemaAdapterFactory` and related components for Parquet scanning. DataFusion now handles schema adaptation differently.","message":"The `SchemaAdapterFactory` has been fully removed from Parquet scanning. This includes the `SchemaAdapter`, `SchemaMapper`, `DefaultSchemaAdapterFactory` traits/structs.","severity":"deprecated","affected_versions":">= 49.0.0 (deprecated in 49.0.0, removed later)"},{"fix":"Be aware of potential performance implications due to statistics collection on table registration. If undesired, explicitly set `ctx.session_config().with_collect_statistics(False)` or configure via `config.set('datafusion.execution.collect_statistics', 'false')`.","message":"The default value of the `datafusion.execution.collect_statistics` configuration setting changed from `false` to `true`. This means DataFusion will now collect and store statistics by default when a table is first created via `CREATE EXTERNAL TABLE` or DataFrame `register_*` APIs.","severity":"gotcha","affected_versions":">= 48.0.0"},{"fix":"Update custom UDF implementations to utilize `FieldRef` where type and nullability information is accessed.","message":"For advanced User-Defined Functions (UDFs), `UDF` traits now use `FieldRef` rather than `DataType` and nullability directly. `FieldRef` provides access to metadata fields, supporting extension types.","severity":"breaking","affected_versions":">= 48.0.0"}],"env_vars":null,"search_vec":"'52.3.0':78 'activ':71 'apach':1,12,95 'api':36 'arrow':13,96 'bind':9 'build':24 'copi':63 'core':86 'csv':42 'current':75 'data':39,49,64,89 'datafram':35,94 'datafus':2,19,87 'effici':60 'enabl':21 'engin':18,56,92 'etl':98 'exchang':65 'execut':26 'focus':58 'high':28 'high-perform':27 'in-memori':14,46 'includ':41 'json':44 'leverag':50 'librari':6,69 'maintain':72 'memori':16,48 'parquet':43 'perform':29 'process':90 'project':88 'provid':8 'pyarrow':67 'python':3,5 'queri':17,30,55,91 'releas':81 'rust':53,97 'rust-written':52 'sourc':40 'sql':32,93 'sync':83 'typic':80 'use':31 'user':22 'various':38 'version':76 'written':54 'zero':62 'zero-copi':61","created_at":"2026-04-11T09:13:16.772522+00:00","updated_at":"2026-04-16T05:30:34.999630+00:00","problems":[{"fix":"Run `pip install datafusion` in your terminal to install the package.","cause":"The datafusion Python package has not been installed in the current Python environment or the environment is not active.","error":"ModuleNotFoundError: No module named 'datafusion'"},{"fix":"Ensure consistent schema definitions across joined tables, explicitly cast columns to matching types if necessary, and verify that Pandas DataFrame dtypes are compatible with PyArrow/DataFusion before conversion.","cause":"This error often arises when joining DataFrames or converting data, particularly from Pandas, due to subtle differences in schema metadata (e.g., nullability, specific types, or internal Arrow metadata) that prevent DataFusion from successfully optimizing or executing the join plan.","error":"DataFusion error: Internal(\"PhysicalOptimizer rule 'join_selection' failed. Schema mismatch.\")"},{"fix":"Explicitly select and alias any duplicate column names in your SQL query or DataFrame API operation to ensure unique qualified field names in the resulting schema.","cause":"When performing a join operation and implicitly selecting all columns (e.g., `SELECT *`), if both joined tables contain columns with identical names, DataFusion's planner encounters ambiguity and reports duplicate qualified field names.","error":"DataFusion error: Plan(\"Schema contains duplicate qualified field name '...' \")"},{"fix":"Simplify the SQL query by breaking down complex subqueries into multiple, simpler steps, or rewrite the query using supported DataFusion DataFrame API operations if a direct SQL translation is not working.","cause":"DataFusion's SQL planner does not yet support all complex SQL operators or patterns within subqueries, leading to a planning error.","error":"DataFusion error: Error during planning: Unsupported operator in the subquery plan."},{"fix":"Inspect the schema of the struct columns and, if necessary, cast the fields within the struct to compatible or simpler types (e.g., string) before attempting the `to_pandas()` conversion.","cause":"This error can occur when converting a DataFusion DataFrame, especially one containing complex types like struct columns, to a Pandas DataFrame using `to_pandas()`, if the underlying Arrow schema of the struct fields does not align with Pandas' expectations.","error":"ArrowInvalid: Schema at index 0 was different"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"54.0.0","cli_name":"datafusion","cli_version":"sh: 1: datafusion: not found","type":"library","homepage":"https://datafusion.apache.org/python","github":"https://github.com/apache/datafusion-python","docs":"https://datafusion.apache.org/python","changelog":null,"pypi":"https://pypi.org/project/datafusion/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","database","serialization"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-28","next_check":"2026-07-28","install_tag":null}}