{"id":1453,"library":"distributed","title":"Dask Distributed","description":"Distributed is the scheduler for Dask, providing a distributed computation engine for parallel and out-of-core analytics. It manages computations across a cluster of machines, featuring dynamic task scheduling, fault tolerance, and diagnostics. The library is actively maintained with frequent releases, typically on a monthly or bi-monthly schedule, following a `YYYY.MM.patch` versioning scheme. The current version is 2026.3.0.","status":"active","version":"2026.3.0","language":"python","source_language":"en","source_url":"https://github.com/dask/distributed","tags":["dask","distributed computing","parallelism","async","scheduler","data science","big data"],"install":[{"cmd":"pip install \"distributed[complete]\"","lang":"bash","label":"Recommended full installation"},{"cmd":"pip install distributed","lang":"bash","label":"Minimal installation"}],"dependencies":[{"reason":"Core dependency for Dask's array, dataframe, etc. objects.","package":"dask","optional":false},{"reason":"Required for certain data serialization and I/O operations (>=16 for recent versions).","package":"pyarrow","optional":false},{"reason":"Required to run the Dask diagnostics dashboard.","package":"bokeh","optional":true},{"reason":"Utility library (>=0.12.0 for recent versions).","package":"toolz","optional":false}],"imports":[{"note":"While `dask.distributed` is a valid module, `distributed` is the top-level package and preferred import path for its core components.","wrong":"from dask.distributed import Client","symbol":"Client","correct":"from distributed import Client"},{"symbol":"LocalCluster","correct":"from distributed import LocalCluster"},{"symbol":"fire_and_forget","correct":"from distributed import fire_and_forget"}],"quickstart":{"code":"from distributed import Client, LocalCluster\nimport time\n\ndef inc(x):\n    time.sleep(0.1) # Simulate work\n    return x + 1\n\n# Start a local Dask cluster\n# Using 'with' statement ensures proper cleanup\nwith LocalCluster(n_workers=2, threads_per_worker=2, dashboard_address=':8787') as cluster:\n    print(f\"Dashboard available at: {cluster.dashboard_link}\")\n\n    # Connect a client to the cluster\n    with Client(cluster) as client:\n        print(f\"Client connected to: {client.scheduler.address}\")\n\n        # Submit tasks\n        futures = client.map(inc, range(10))\n\n        # Gather results\n        results = client.gather(futures)\n\n        print(f\"Results: {results}\")\n        assert results == [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n\n    print(\"Client disconnected.\")\nprint(\"LocalCluster closed.\")","lang":"python","description":"This quickstart demonstrates how to set up a local Dask cluster using `LocalCluster` and connect a `Client` to it. It then submits a simple `inc` function for parallel execution and gathers the results. The use of context managers (`with`) is crucial for proper resource management and cleanup. A dashboard link is printed if `bokeh` is installed."},"warnings":[{"fix":"Upgrade `pyarrow` to version 16 or newer: `pip install 'pyarrow>=16'`.","message":"Starting with version 2026.1.2, `distributed` requires `pyarrow>=16`. Older versions of `pyarrow` may lead to `ImportError` or serialization issues, especially with Dask DataFrames.","severity":"breaking","affected_versions":"2026.1.2+"},{"fix":"Ensure your Python environment is version 3.10 or greater. Upgrade Python or use a compatible environment.","message":"`distributed` now requires Python 3.10 or newer. Support for Python 3.9 and older has been dropped.","severity":"breaking","affected_versions":"2025.12.0+"},{"fix":"Upgrade `toolz` to version 0.12.0 or newer: `pip install 'toolz>=0.12.0'`.","message":"Starting with version 2025.12.0, `distributed` requires `toolz>=0.12.0`. Older versions may cause compatibility errors.","severity":"breaking","affected_versions":"2025.12.0+"},{"fix":"Install `distributed` with the `[dashboard]` or `[complete]` extra: `pip install \"distributed[dashboard]\"`.","message":"The Dask diagnostics dashboard requires `bokeh` to be installed. Without it, the dashboard link will still be provided but will not function correctly, often leading to a blank page or errors.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Wrap `Client` and `LocalCluster` instantiation in `with` statements: `with LocalCluster(...) as cluster: ... with Client(cluster) as client: ...`.","message":"When using `Client` or `LocalCluster`, always use them as context managers (`with Client(...) as client:`) or explicitly call their `.close()` methods to ensure proper cleanup of resources, especially in scripts or long-running applications. Failing to do so can leave orphaned processes or open ports.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Use a currently supported Python version (e.g., 3.10-3.12) or await an official `distributed` release with Python 3.13 compatibility.","message":"`distributed` might not yet be fully compatible with Python 3.13. Running on this unverified Python version can lead to unexpected behavior, including process timeouts or failures during initialization. Please check `distributed`'s official documentation for current Python compatibility.","severity":"breaking","affected_versions":"All versions (until Python 3.13 is officially supported)"}],"env_vars":null,"search_vec":"'2026.3.0':64 'across':25 'activ':41 'analyt':21 'async':69 'bi':52 'bi-month':51 'big':73 'cluster':27 'comput':12,24,67 'core':20 'current':61 'dask':1,8,65 'data':71,74 'diagnost':37 'distribut':2,3,11,66 'dynam':31 'engin':13 'fault':34 'featur':30 'follow':55 'frequent':44 'librari':39 'machin':29 'maintain':42 'manag':23 'month':49,53 'out-of-cor':17 'parallel':15,68 'provid':9 'releas':45 'schedul':6,33,54,70 'scheme':59 'scienc':72 'task':32 'toler':35 'typic':46 'version':58,62 'yyyy.mm.patch':57","created_at":"2026-04-09T03:48:35.287073+00:00","updated_at":"2026-04-16T14:26:43.056226+00:00","problems":[{"fix":"Install the 'distributed' library using pip (`pip install distributed`) or conda (`conda install distributed`). If using a virtual environment, ensure it is activated before running your script.","cause":"The 'distributed' library, which is part of Dask, is not installed or is not accessible in the Python environment where the code is being executed. This can also occur due to an inactive virtual environment or incorrect PYTHONPATH settings.","error":"ModuleNotFoundError: No module named 'distributed'"},{"fix":"Verify that the Dask scheduler is running and that its address (host and port) matches what the client is trying to connect to. Check firewall configurations or security groups if the scheduler is on a remote machine.","cause":"The Dask client is unable to establish a connection with the Dask scheduler. This commonly happens if the scheduler is not running, is configured with an incorrect host or port, or if a firewall is blocking the connection between the client and the scheduler.","error":"ConnectionRefusedError: [Errno 111] Connection refused"},{"fix":"Reduce the memory footprint of your tasks, increase the 'memory_limit' parameter when initializing your Dask client or workers (e.g., `Client(memory_limit='8GB')`), or decrease the number of workers to allocate more memory to each.","cause":"A Dask worker has consumed nearly all of its allocated memory, leading the Dask Nanny process to automatically restart it to prevent system instability. This usually indicates that tasks are too memory-intensive for the worker's current configuration or that there's a memory leak in the user's code.","error":"distributed.worker - WARNING - Worker exceeded 95% memory budget. Restarting..."},{"fix":"Implement robust error handling for Dask futures, manage explicit timeouts using `Future.result(timeout=...)` or `distributed.wait(futures, timeout=...)`, and review worker/scheduler logs for underlying causes of unexpected cancellations. Adjust scheduler timeout configurations in cloud environments if 'idle timeouts' are the issue.","cause":"A Dask future or task was explicitly cancelled, timed out, or implicitly cancelled due to a failure or disconnection of a scheduler or worker it depended on. In cloud deployments, an 'idle timeout' on the scheduler can also trigger this error.","error":"CancelledError"},{"fix":"Upgrade your Python installation to version 3.9.2 or newer, or use a Python version older than 3.9.0.","cause":"This specific TypeError was a known compatibility issue with `distributed` when used with Python versions 3.9.0 and 3.9.1, stemming from internal changes in Python's core.","error":"TypeError: unhashable type: 'list' (when importing distributed)"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"2026.8.0","cli_name":"dask-scheduler","cli_version":"dask-scheduler, version 2026.3.0","type":"library","homepage":"https://distributed.dask.org","github":"https://github.com/dask/distributed","docs":null,"changelog":null,"pypi":"https://pypi.org/project/distributed/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","workflow","devops"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-27","next_check":"2026-07-28","install_tag":"verified"}}