{"id":4622,"library":"loky","title":"Loky - Robust Process Pool Executor","description":"Loky provides a robust, cross-platform, and cross-version implementation of Python's `concurrent.futures.ProcessPoolExecutor`. It enhances multiprocessing by offering reusable executors, transparent `cloudpickle` integration for complex object serialization, and deadlock-free process management, addressing common pitfalls in parallel Python computing. The library is actively maintained, with its current version being 3.5.6. It primarily follows a minor release cadence with bug fixes and improvements.","status":"active","version":"3.5.6","language":"python","source_language":"en","source_url":"https://github.com/joblib/loky","tags":["multiprocessing","concurrency","process pool","executor","joblib","cloudpickle"],"install":[{"cmd":"pip install loky","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Requires Python 3.9 or higher for the latest versions.","package":"python","optional":false},{"reason":"Optional dependency that enables serialization of a wider range of objects (e.g., lambda functions, interactively defined functions), especially those in the __main__ module, avoiding common pickling errors.","package":"cloudpickle","optional":true},{"reason":"Optional dependency used for early detection of memory leaks in worker processes.","package":"psutil","optional":true}],"imports":[{"note":"This is the recommended entry point for most common use cases, providing a managed, reusable process pool.","symbol":"get_reusable_executor","correct":"from loky import get_reusable_executor"},{"note":"This provides a direct, robust replacement for `concurrent.futures.ProcessPoolExecutor` with enhanced error handling.","symbol":"ProcessPoolExecutor","correct":"from loky import ProcessPoolExecutor"},{"note":"Loky has its own `set_start_method` function which *must* be used instead of `multiprocessing.set_start_method` to ensure proper behavior and compatibility with Loky's internal process management.","wrong":"import multiprocessing; multiprocessing.set_start_method('spawn')","symbol":"set_start_method","correct":"from loky import set_start_method"}],"quickstart":{"code":"import os\nfrom loky import get_reusable_executor\n\ndef worker_function(x):\n    # Simulate some work\n    pid = os.getpid()\n    return f\"Processed {x} by PID {pid}\"\n\nif __name__ == \"__main__\":\n    # Using get_reusable_executor for managed process pool\n    with get_reusable_executor(max_workers=2) as executor:\n        results = list(executor.map(worker_function, range(5)))\n    print(results)\n\n    # Direct ProcessPoolExecutor usage (similar to concurrent.futures)\n    from loky import ProcessPoolExecutor\n    with ProcessPoolExecutor(max_workers=2) as executor:\n        results_direct = list(executor.map(worker_function, range(5, 10)))\n    print(results_direct)","lang":"python","description":"This quickstart demonstrates the two primary ways to use Loky: `get_reusable_executor()` for a managed and persistent pool, and `ProcessPoolExecutor()` for a direct `concurrent.futures`-like experience. The `if __name__ == \"__main__\":` block is included for robust multiprocessing execution across different operating systems."},"warnings":[{"fix":"Always import and use `loky.set_start_method()` when configuring the process start method for Loky executors.","message":"Loky's `set_start_method` is incompatible with `multiprocessing.set_start_method`. Attempting to use the standard library's function will not configure Loky's process startup correctly and can lead to unexpected behavior or errors.","severity":"breaking","affected_versions":"All versions"},{"fix":"For scenarios requiring strict process termination and resource cleanup, consider if `get_reusable_executor()`'s persistence model is suitable. If not, explicitly manage process lifecycle and ensure all resources are released or cleaned up outside the main process before attempting directory changes or deletions.","message":"When using `loky.get_reusable_executor()` on Windows, worker processes are kept alive for reuse, which can prevent cleanup operations (e.g., `os.chdir()` or deleting temporary directories) if they were used within the worker context. Even `loky.ProcessPoolExecutor()` may leave one process active after explicit shutdown.","severity":"gotcha","affected_versions":"All versions, particularly on Windows"},{"fix":"For performance-critical applications, profile the serialization overhead. If it's a bottleneck, optimize objects to be standard-picklable or explore custom reducers for specific data types to minimize `cloudpickle`'s impact. Consider setting `LOKY_PICKLER=pickle` if `cloudpickle`'s broader serialization capabilities are not needed.","message":"While `loky` transparently integrates `cloudpickle` to serialize non-picklable objects, this serialization can introduce performance overhead compared to Python's standard `pickle` module, especially for very large objects or high-frequency task submission.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Be aware of the `fork+exec` behavior. If you encounter issues with shared memory or inherited resources, ensure all necessary state is explicitly passed to worker processes rather than relying on implicit inheritance. If working with very old Python versions or specific `multiprocessing.Pool` expectations, consider how this difference might affect your application.","message":"Loky on POSIX systems (e.g., Linux, macOS) uses `fork+exec` for all processes to ensure consistent and robust spawn behavior, which is safer when interacting with third-party libraries (e.g., OpenMP, macOS Accelerate) compared to `multiprocessing.Pool`'s default `fork` (or pre-Python 3.8 macOS `fork`). This difference might subtly alter behavior if your code relies on `fork` without `exec` semantics.","severity":"gotcha","affected_versions":"All versions on POSIX systems"}],"env_vars":null,"search_vec":"'3.5.6':59 'activ':52 'address':42 'bug':68 'cadenc':66 'cloudpickl':30,78 'common':43 'complex':33 'comput':48 'concurr':73 'concurrent.futures.processpoolexecutor':21 'cross':11,15 'cross-platform':10 'cross-vers':14 'current':56 'deadlock':38 'deadlock-fre':37 'enhanc':23 'executor':5,28,76 'fix':69 'follow':62 'free':39 'implement':17 'improv':71 'integr':31 'joblib':77 'librari':50 'loki':1,6 'maintain':53 'manag':41 'minor':64 'multiprocess':24,72 'object':34 'offer':26 'parallel':46 'pitfal':44 'platform':12 'pool':4,75 'primarili':61 'process':3,40,74 'provid':7 'python':19,47 'releas':65 'reusabl':27 'robust':2,9 'serial':35 'transpar':29 'version':16,57","created_at":"2026-04-12T14:00:31.720688+00:00","updated_at":"2026-04-16T16:22:09.284903+00:00","problems":[{"fix":"Wrap the code that creates and uses the `loky` executor (or any multiprocessing code) within an `if __name__ == '__main__':` block.","cause":"This error occurs when attempting to create new processes in a Python script without protecting the entry point, especially on Windows and macOS where the default process start method is 'spawn' and the main module is re-imported by child processes.","error":"RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase."},{"fix":"Ensure that all arguments to functions executed by the `loky` executor, as well as the function itself and any objects it closes over, are picklable. For complex objects, you might need to implement `__reduce__` or use a more powerful serialization library like `dill` by configuring `loky` to use it via `loky.set_loky_pickler('dill')`.","cause":"Objects and functions passed between processes via `loky`'s `ProcessPoolExecutor` must be 'picklable' (serializable). This error occurs when an unpicklable object, such as a local function, a lambda, or a complex object with unpicklable attributes (like a `_thread.lock` or `weakref`), is implicitly or explicitly sent to a worker process.","error":"TypeError: cannot pickle '...' object"},{"fix":"Avoid nesting parallel calls. If using `joblib`, ensure that `Parallel` is not called from within a function that is itself being executed by another multiprocessing pool. Consider refactoring your code to flatten the parallelism or use thread-based parallelism for inner loops if appropriate.","cause":"This warning typically arises when `joblib` (which uses `loky` as a backend) is invoked within an already existing multiprocessing context, indicating that nested parallelism might not be correctly handled or that the inner `loky` pool is effectively disabled or limited to a single job.","error":"Loky-backed parallel loops cannot be called in a multiprocessing, setting n_jobs=1 warning"},{"fix":"Similar to `TypeError: cannot pickle`, meticulously review the function arguments and any objects referenced by the function that are being sent to `loky`'s worker processes to ensure they are all picklable. If `cloudpickle` (loky's default advanced pickler) is still failing, consider simplifying the objects or explicitly handling their serialization.","cause":"This error occurs when a worker process fails to deserialize a task or its arguments, usually due to an unpicklable object being passed as part of the task, or a corruption in the serialization stream, leading to the worker process crashing or becoming unresponsive.","error":"BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable."}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"3.5.6","cli_name":"","cli_version":null,"type":"library","homepage":"http://loky.readthedocs.io","github":"https://github.com/joblib/loky","docs":null,"changelog":null,"pypi":"https://pypi.org/project/loky/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","serialization","workflow"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-30","next_check":"2026-07-28","install_tag":null}}