{"id":1442,"library":"dask-expr","title":"Dask Expressions","description":"Dask-expr provides a high-level expression system for Dask DataFrames, focusing on query optimization and improved organization. It became the default backend for `dask.dataframe` since Dask version 2024.3.0. The library, currently at version 2.0.0 (released January 21, 2025), is primarily maintained as part of the main Dask project, with its separate GitHub repository no longer actively maintained.","status":"active","version":"2.0.0","language":"python","source_language":"en","source_url":"https://github.com/dask/dask","tags":["dask","dataframe","query-optimization","distributed-computing","big-data"],"install":[{"cmd":"pip install dask-expr","lang":"bash","label":"PyPI"},{"cmd":"conda install conda-forge::dask-expr","lang":"bash","label":"Conda-forge"}],"dependencies":[{"reason":"Core Dask functionality, dask-expr is the default backend for dask.dataframe.","package":"dask","optional":false},{"reason":"Dask-expr requires pandas version 2.0 or higher.","package":"pandas","optional":false},{"reason":"Often used for efficient data interchange and I/O with Parquet files.","package":"pyarrow","optional":true},{"reason":"For reading/writing from various file systems (e.g., S3, GCS).","package":"fsspec","optional":true}],"imports":[{"note":"While functional, dask-expr is now the default backend for dask.dataframe. It is often not explicitly imported for basic DataFrame operations.","symbol":"dask_expr","correct":"import dask_expr as dx"},{"note":"The expression system is implicitly used when working with dask.dataframe since Dask 2024.3.0.","symbol":"dask.dataframe","correct":"import dask.dataframe as dd"}],"quickstart":{"code":"import dask.dataframe as dd\n\n# Create a Dask DataFrame (internally uses the dask-expr system)\ndf = dd.from_dict({'a': range(1000), 'b': [f'cat_{i%5}' for i in range(1000)]}, npartitions=4)\n\n# Perform some operations\nresult = df.groupby('b')['a'].mean()\n\n# Compute the result\nprint(result.compute())\n\n# To see the optimized query plan (requires graphviz to be installed)\n# try:\n#    df.optimize().explain()\n# except ImportError:\n#    print(\"Install graphviz to visualize the query plan: pip install 'graphviz'\")","lang":"python","description":"This quickstart demonstrates how to use Dask DataFrame, which leverages the dask-expr expression system internally for query optimization since Dask version 2024.3.0. No explicit `dask_expr` import is typically needed for standard DataFrame operations."},"warnings":[{"fix":"Ensure all dependencies are compatible with `pandas>=2`. You may need to upgrade or constrain other libraries if conflicts arise. Consider using isolated environments (conda, virtualenv) for different projects.","message":"The `dask-expr` library, particularly versions 0.5.1 and above, requires `pandas>=2`. This can lead to dependency conflicts if other libraries in your environment pin `pandas` to an older version (e.g., `<2`).","severity":"breaking","affected_versions":">=0.5.1"},{"fix":"For contributions or detailed technical understanding, refer to the Dask main repository and documentation, specifically the `dask.dataframe` sections. Continue to install `dask-expr` if your Dask version is older than 2024.3.0 and you wish to use the query planning features. Otherwise, it is installed by default with recent Dask versions.","message":"The `dask-expr` GitHub repository is no longer actively maintained, as its implementation has been moved into the main `dask/dask` repository and it's now the default backend for `dask.dataframe`. While the PyPI package still exists, new development and core maintenance happen within the Dask project itself.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Use `persist()` sparingly and only when absolutely necessary, or when the full dataset is genuinely needed for subsequent complex operations. Re-evaluate if `persist()` is truly required, as the optimizer often handles intermediate computations efficiently without explicit persistence.","message":"Using `df.persist()` with the dask-expr query optimizer can sometimes block optimizations like projection or filter pushdown into the I/O layer.","severity":"gotcha","affected_versions":"All versions where dask-expr is enabled"},{"fix":"If 'named GroupBy Aggregations' are critical for your workflow, you might need to structure your aggregations differently (e.g., performing multiple individual aggregations and then combining them) or temporarily opt-out of the dask-expr backend if using an older Dask version where it wasn't the default (`dask.config.set({'dataframe.query-planning': False})`).","message":"Dask-expr does not currently support 'named GroupBy Aggregations', which is a feature available in the legacy Dask DataFrame API.","severity":"gotcha","affected_versions":"All versions where dask-expr is enabled"},{"fix":"Ensure `numpy` is installed in your environment. It is typically installed as a dependency when installing `dask` or `dask[array]`. You can explicitly install it via `pip install numpy` or `conda install numpy`.","message":"Dask (and by extension, `dask.dataframe`) has `numpy` as a fundamental dependency. If `numpy` is not installed, importing `dask.dataframe` will fail with an `ImportError`.","severity":"breaking","affected_versions":"All versions"}],"env_vars":null,"search_vec":"'2.0.0':39 '2024.3.0':33 '2025':43 '21':42 'activ':61 'backend':27 'becam':24 'big':72 'big-data':71 'comput':70 'current':36 'dask':1,4,14,31,52,63 'dask-expr':3 'dask.dataframe':29 'data':73 'datafram':15,64 'default':26 'distribut':69 'distributed-comput':68 'expr':5 'express':2,11 'focus':16 'github':57 'high':9 'high-level':8 'improv':21 'januari':41 'level':10 'librari':35 'longer':60 'main':51 'maintain':46,62 'optim':19,67 'organ':22 'part':48 'primarili':45 'project':53 'provid':6 'queri':18,66 'query-optim':65 'releas':40 'repositori':58 'separ':56 'sinc':30 'system':12 'version':32,38","created_at":"2026-04-09T03:48:07.137220+00:00","updated_at":"2026-04-16T05:10:25.776239+00:00","problems":[{"fix":"Install `dask-expr` using `pip install dask-expr` or ensure a comprehensive Dask installation with `pip install \"dask[dataframe]\"` or `conda install dask-expr -c conda-forge`.","cause":"`dask-expr` is not installed in the environment. Even though `dask.dataframe` became the default backend for `dask-expr` in Dask 2024.3.0, it still needs to be explicitly installed as a separate package if not pulled by `dask[dataframe]` installation.","error":"ModuleNotFoundError: No module named 'dask_expr'"},{"fix":"This is typically a compatibility issue between `dask-sql` (or another library) and the current `dask` / `dask-expr` versions. Check the documentation or GitHub issues for `dask-sql` (or the relevant library) for updated compatibility information or try installing a specific older version of `dask-sql` that is known to be compatible with your Dask/dask-expr setup.","cause":"This error often occurs when using `dask-sql` or other dependent libraries with newer versions of Dask and `dask-expr`. It indicates that these libraries are trying to import a module (`dask_expr.io`) that has been moved, removed, or has changed its internal structure, leading to compatibility issues.","error":"ModuleNotFoundError: No module named 'dask_expr.io'"},{"fix":"Consult the `dask-expr` API coverage documentation (often linked in the error message itself) to confirm support for the specific function. If it's not supported, consider refactoring your code to use alternative, supported operations. As a temporary workaround, you can disable `dask-expr` query planning with `import dask; dask.config.set({'dataframe.query-planning': False})`.","cause":"This `AttributeError` (or similar ones indicating an unsupported API function) means you are using a Dask DataFrame API method or attribute that is not yet fully implemented or has different behavior under the `dask-expr` backend.","error":"AttributeError: 'Min' object has no attribute 'dtype'"},{"fix":"To enable the new backend and remove the warning, install `dask-expr` (`pip install dask-expr` or `pip install \"dask[dataframe]\"`) and explicitly enable query planning: `import dask; dask.config.set({'dataframe.query-planning': True})`. If you wish to suppress the warning without transitioning immediately, use `dask.config.set({'dataframe.query-planning-warning': False})`.","cause":"This warning signals that the legacy Dask DataFrame backend is being replaced by `dask-expr`. It appears when `dask.dataframe` is imported, prompting users to install `dask-expr` and enable query planning to transition to the new, optimized backend.","error":"DeprecationWarning: The current Dask DataFrame implementation is deprecated."}],"ecosystem":"pypi","meta_description":null,"install_score":95,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"2.0.0","cli_name":"","cli_version":null,"type":"library","homepage":"https://www.dask.org","github":"https://github.com/dask-contrib/dask-expr","docs":null,"changelog":null,"pypi":"https://pypi.org/project/dask-expr/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-06-28","next_check":"2026-07-28","install_tag":"verified"}}