{"id":3476,"library":"dvc-task","title":"dvc-task","description":"dvc-task is an extensible Python library for queuing, running, and managing background jobs (processes) from standalone applications. It is built on Celery but leverages a pre-configured filesystem transport, eliminating the need for a full AMQP messaging server. As of April 2026, the current version is 0.40.2, with active development and somewhat frequent, though irregular, releases.","status":"active","version":"0.40.2","language":"python","source_language":"en","source_url":"https://github.com/iterative/dvc-task","tags":["task queue","celery","dvc","background jobs","async"],"install":[{"cmd":"pip install dvc-task","lang":"bash","label":"Install dvc-task"}],"dependencies":[{"reason":"Core task queue framework, dvc-task is built on top of it. Requires version 5.3 or later.","package":"celery"},{"reason":"Messaging library used by Celery, requires version 5.3 or later.","package":"kombu"}],"imports":[{"note":"Main application class for a filesystem-based Celery app.","symbol":"FSApp","correct":"from dvc_task.app import FSApp"},{"note":"Manages background processes within tasks.","symbol":"ProcessManager","correct":"from dvc_task.proc import ProcessManager"}],"quickstart":{"code":"from dvc_task.app import FSApp\nfrom dvc_task.proc import ProcessManager\nimport time\nimport os\n\n# Create a filesystem-based Celery app\n# For a real application, consider a more persistent wdir\napp = FSApp(wdir=\"./dvc-task-data\")\n\n@app.task\ndef run_command_task(command_args, task_name):\n    manager = ProcessManager(wdir=os.path.join(app.wdir, 'processes'))\n    # Run a simple command as a background process\n    # The '.delay()' call queues the task\n    signature = manager.run(command_args, name=task_name).delay()\n    return signature.id\n\n# Example usage:\nif __name__ == \"__main__\":\n    print(\"Starting dvc-task example...\")\n    task_id = run_command_task.delay(command_args=[\"echo\", \"Hello, dvc-task!\"], task_name=\"greet_task\")\n    print(f\"Task 'greet_task' queued with ID: {task_id}\")\n\n    print(\"Starting temporary worker...\")\n    # In a real scenario, the worker would run in a separate process/daemon\n    # For quickstart, we run a temporary worker that exits when the queue is empty\n    worker = app.TemporaryWorker()\n    worker.start() # This blocks until the queue is empty or timeout\n\n    # Clean up (optional, but good for quickstart)\n    app.clean()\n    print(\"dvc-task example finished and cleaned up.\")\n","lang":"python","description":"This quickstart demonstrates how to initialize a filesystem-based Celery application (FSApp), define a task to run a background process using ProcessManager, and then start a temporary worker to process the task. It showcases how to queue a simple 'echo' command."},"warnings":[{"fix":"Upgrade Celery and Kombu to at least version 5.3.0: `pip install 'celery>=5.3.0' 'kombu>=5.3.0'`.","message":"Version 0.3.0 introduced a breaking change by bumping the minimum required versions for Celery and Kombu to 5.3.0. Users on older versions of these dependencies will need to upgrade.","severity":"breaking","affected_versions":"<0.3.0"},{"fix":"Upgrade to dvc-task version 0.40.1 or later to resolve Celery Monitor Shutdown issues.","message":"Users observed 'Celery Monitor Shutdowns' prior to version 0.40.1, indicating potential instability or unexpected behavior in monitoring. Version 0.40.1 includes a fix (dvc-10427) for this issue.","severity":"gotcha","affected_versions":"<0.40.1"},{"fix":"Be aware of potential platform-specific issues on Windows. Refer to Celery's documentation for known Windows limitations and workarounds.","message":"While `dvc-task` is tested on Windows, its upstream dependency Celery is not officially supported on Windows. This means that certain issues specific to Windows might stem from Celery itself and could have limited official support.","severity":"gotcha","affected_versions":"All"},{"fix":"Carefully review the changelog between `0.4.0` and `0.40.1` for unlisted breaking changes or behavioral differences, and thoroughly test your application after upgrading.","message":"The version numbering jumped from `0.4.0` to `0.40.1`. While not explicitly stated as breaking, such a large jump in the minor version often signals significant changes or potential incompatibilities, even if release notes are brief. Users should review the changelog carefully when upgrading across this boundary.","severity":"gotcha","affected_versions":"All versions before 0.40.1 when upgrading to 0.40.1+"}],"env_vars":null,"search_vec":"'0.40.2':53 '2026':48 'activ':55 'amqp':42 'applic':22 'april':47 'async':69 'background':17,67 'built':25 'celeri':27,65 'configur':33 'current':50 'develop':56 'dvc':2,5,66 'dvc-task':1,4 'elimin':36 'extens':9 'filesystem':34 'frequent':59 'full':41 'irregular':61 'job':18,68 'leverag':29 'librari':11 'manag':16 'messag':43 'need':38 'pre':32 'pre-configur':31 'process':19 'python':10 'queu':13 'queue':64 'releas':62 'run':14 'server':44 'somewhat':58 'standalon':21 'task':3,6,63 'though':60 'transport':35 'version':51","created_at":"2026-04-11T17:31:15.518624+00:00","updated_at":"2026-04-16T14:44:16.476762+00:00","problems":[{"fix":"Downgrade `fsspec` to a version compatible with your `dvc` and `dvc-task` installation, for example, `pip install 'fsspec==2022.11.0'` or upgrade `dvc` and `dvc-task` to their latest versions to ensure `fsspec` compatibility.","cause":"This error occurs due to an incompatibility between the `dvc` and `dvc-task` libraries and their dependency, `fsspec`.","error":"ImportError: cannot import name 'fsspec_loop' from 'fsspec.asyn'"},{"fix":"Ensure all necessary dependencies and environment variables are consistent between where the task was queued and where the worker is executing it. Debug the failing task by trying to run it outside the queue (`dvc exp run` without `--queue`). Running `dvc exp clean` can also help clear any corrupted temporary files.","cause":"This indicates an `AttributeError` occurring within a `dvc-task` worker, preventing the execution of a queued job, possibly due to inconsistent environments or internal state corruption. The full traceback in the logs (e.g., in `.dvc/tmp/exps/celery/result` files) would provide the specific attribute issue.","error":"Queued jobs failing before they get started (often with `AttributeError` in logs)"},{"fix":"Downgrade `dvc-objects` to a compatible version (e.g., `pip install 'dvc-objects<=1.3.2'`) or upgrade your entire `dvc` package to its latest version to resolve the dependency conflict.","cause":"This `ImportError` signifies a version mismatch between your `dvc` installation (which includes `dvc-task` as a subproject) and the `dvc_objects` library.","error":"ERROR: unexpected error - cannot import name 'umask' from 'dvc_objects.fs.system'"},{"fix":"Clear the corrupted status by manually deleting the temporary experiment directory (`.dvc/tmp/exps`) or running the command `dvc exp clean`.","cause":"This error points to a parsing issue, likely due to corrupted internal JSON or structured data files used by `dvc-task` to manage the experiment queue status. This can occur from abrupt process termination or concurrency issues.","error":"ERROR: unexpected error - Extra data: line 1 column 56 (char 55). (from `dvc queue status`)"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.40.2","cli_name":"dvc-task","cli_version":"sh: 1: dvc-task: not found","type":"library","homepage":null,"github":"https://github.com/iterative/dvc-task","docs":null,"changelog":null,"pypi":"https://pypi.org/project/dvc-task/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["workflow","database"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-29","next_check":"2026-07-28","install_tag":null}}