{"id":3333,"library":"luigi","title":"Luigi Workflow Management","description":"Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, and much more. It's developed by Spotify and is currently in version 3.8.0, with minor releases typically occurring every few months.","status":"active","version":"3.8.0","language":"python","source_language":"en","source_url":"https://github.com/spotify/luigi","tags":["workflow-orchestration","ETL","task-scheduling","data-pipeline","batch-processing"],"install":[{"cmd":"pip install luigi","lang":"bash","label":"Install Luigi"}],"dependencies":[],"imports":[{"symbol":"Task","correct":"import luigi\n\nclass MyTask(luigi.Task): ..."},{"symbol":"Parameter","correct":"import luigi\n\nclass MyTask(luigi.Task):\n    param = luigi.Parameter()"},{"symbol":"LocalTarget","correct":"import luigi\n\nluigi.LocalTarget('path/to/file')"},{"symbol":"build","correct":"import luigi\n\nluigi.build([my_task_instance], local_scheduler=True)"}],"quickstart":{"code":"import luigi\nimport datetime\nimport os\n\nclass GenerateReport(luigi.Task):\n    date = luigi.DateParameter(default=datetime.date.today())\n\n    def run(self):\n        # Simulate some data processing\n        report_content = f\"Daily Report for {self.date.isoformat()}\\n\" \\\n                         f\"Generated by Luigi.\\n\"\n\n        # Write the report to a target file\n        with self.output().open('w') as f:\n            f.write(report_content)\n\n    def output(self):\n        # Define where the output of this task will be stored\n        # Using an environment variable for flexibility or defaulting to current dir\n        output_dir = os.environ.get('LUIGI_OUTPUT_DIR', '.')\n        return luigi.LocalTarget(os.path.join(output_dir, f'report_{self.date.isoformat()}.txt'))\n\nif __name__ == '__main__':\n    # To run this task using the command line (most common):\n    # 1. Start the Luigi scheduler daemon in a separate terminal: luigid --port 8082\n    # 2. Run your script: python your_script_name.py GenerateReport --date 2023-10-26\n    #    (or omit --date for today's date if default is set)\n    # 3. If you don't want to run luigid, use the --local-scheduler flag:\n    #    python your_script_name.py GenerateReport --local-scheduler\n\n    # For programmatic execution (e.g., in a wrapper script or test):\n    # The 'local_scheduler=True' ensures it runs without an external luigid daemon.\n    luigi.build([GenerateReport(date=datetime.date(2023, 10, 26))], local_scheduler=True)","lang":"python","description":"This quickstart defines a simple Luigi task `GenerateReport` that takes a `date` parameter, simulates generating a report, and writes it to a local file. It demonstrates defining a task, its `run` method, and its `output` target using `luigi.LocalTarget`. The example also shows how to trigger tasks programmatically using `luigi.build` with a local scheduler."},"warnings":[{"fix":"Upgrade your Python environment to a compatible version (3.10, 3.11, 3.12, 3.13) before upgrading Luigi to 3.x.","message":"Luigi 3.x series requires Python >= 3.10 and < 3.14. Projects on older Python 3 versions (e.g., 3.6-3.9) or newer versions (3.14+) will not be compatible.","severity":"breaking","affected_versions":"3.x.x"},{"fix":"Start `luigid` in a separate terminal (`luigid --port 8082`) or always include `--local-scheduler` in your command line execution or `local_scheduler=True` in `luigi.build()` calls.","message":"Luigi tasks require a scheduler to run. By default, it looks for a running `luigid` daemon. For simple local execution or development, remember to use the `--local-scheduler` flag when running tasks from the command line, or `local_scheduler=True` when using `luigi.build()` programmatically.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Review custom plugins or extensions for any direct calls or implicit dependencies on `pkg_resources` and migrate to `importlib.metadata` (for Python 3.8+) or alternative methods.","message":"The `pkg_resources` library, which was historically used for introspection and entry points, has been removed in Luigi 3.7.3. While this is primarily an internal change, it might affect custom plugins or integrations that directly or indirectly relied on Luigi's usage of `pkg_resources`.","severity":"deprecated","affected_versions":">=3.7.3"},{"fix":"Ensure that every `run()` method reliably creates the target specified by its `output()` method. For tasks that don't produce physical outputs, consider using a dummy `LocalTarget('/tmp/luigi_flag_file')` or `luigi.MockTarget` for testing, or ensure their dependencies cover the actual work.","message":"Luigi determines task completion solely based on whether the `output()` targets `exist()`. If a task's `run()` method completes but fails to create its defined `output()`, Luigi will consider it incomplete and re-run it in subsequent invocations. Similarly, manually deleting an output file will cause Luigi to re-run the producing task.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"search_vec":"'3.8.0':38 'batch':16,58 'batch-process':57 'build':12 'complex':13 'current':35 'data':55 'data-pipelin':54 'depend':20 'develop':30 'etl':50 'everi':44 'handl':19 'help':10 'job':17 'luigi':1,4 'manag':3,23 'minor':40 'modul':8 'month':46 'much':26 'occur':43 'orchestr':49 'pipelin':14,56 'process':59 'python':7 'releas':41 'resolut':21 'schedul':53 'spotifi':32 'task':52 'task-schedul':51 'typic':42 'version':37 'visual':24 'workflow':2,22,48 'workflow-orchestr':47","created_at":"2026-04-11T12:36:04.241064+00:00","updated_at":"2026-04-17T14:15:39.246698+00:00","problems":[{"fix":"Upgrade your Python environment to version 3.6 or higher to ensure compatibility with Luigi.","cause":"This error occurs when attempting to run Luigi, which requires Python 3.x, on Python 2.7.","error":"SyntaxError: ('invalid syntax', ('/.pyenv/versions/2.7.14/lib/python2.7/site-packages/luigi/task.py', 145, 21, 'class Task(metaclass=Register):\\n'))"},{"fix":"Ensure that the 'us' module is installed in the correct Python environment by running 'pip install us' in the same environment where Luigi is executed.","cause":"This error indicates that the 'us' module is not found in the Python environment, possibly due to it being installed in a different environment or not installed at all.","error":"ImportError: No module named us"},{"fix":"Set the PYTHONPATH environment variable to include the directory containing 'tasks.py', or run Luigi from the directory where 'tasks.py' is located.","cause":"This error occurs when Luigi cannot locate the 'tasks' module, often due to the module not being in the Python path.","error":"ImportError: No module named tasks"},{"fix":"Avoid using the '--background' option when running 'luigid' on Windows, as it relies on the 'pwd' module.","cause":"The 'pwd' module is specific to Unix-like systems and is not available on Windows, leading to this error when running Luigi on Windows.","error":"ModuleNotFoundError: No module named 'pwd'"},{"fix":"Ensure that all prerequisite tasks are correctly defined and executed before running dependent tasks in Luigi.","cause":"This error indicates that a required dependency task has not been completed successfully before the current task is executed.","error":"RuntimeError: Unfulfilled dependency at run time: Generate_TV_WebScraping_File_X__DataMining_Lu_8213e479cf"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"3.8.1","cli_name":"luigi","cli_version":"No task specified","type":"library","homepage":"https://luigi.readthedocs.io","github":"https://github.com/spotify/luigi","docs":null,"changelog":null,"pypi":"https://pypi.org/project/luigi/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["workflow","data","devops"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-29","next_check":"2026-07-28","install_tag":null}}