{"id":4809,"library":"tqdm-multiprocess","title":"tqdm-multiprocess","description":"tqdm-multiprocess is a Python library that facilitates easy integration of `tqdm` progress bars and logging redirection into multiprocessing workflows. It allows multiple worker processes to display their individual progress bars and log messages cleanly through the main process, along with a global progress bar for aggregate monitoring. The current version is 0.0.11, and its release cadence appears to be feature-driven rather than time-boxed.","status":"active","version":"0.0.11","language":"python","source_language":"en","source_url":"https://github.com/EleutherAI/tqdm-multiprocess","tags":["tqdm","multiprocessing","progress-bar","logging","parallel-processing"],"install":[{"cmd":"pip install tqdm-multiprocess","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"Core functionality for progress bars.","package":"tqdm"},{"reason":"Used for cross-platform terminal coloring, likely for logging output.","package":"colorama"}],"imports":[{"symbol":"TqdmMultiProcessPool","correct":"from tqdm_multiprocess import TqdmMultiProcessPool"}],"quickstart":{"code":"import time\nimport logging\nfrom tqdm_multiprocess import TqdmMultiProcessPool\nfrom tqdm import tqdm\n\nlogging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n\ndef worker_function(task_id, sleep_duration, tqdm_func, global_tqdm):\n    # Initialize a worker-specific tqdm bar\n    worker_bar = tqdm_func(total=10, desc=f'Task {task_id}')\n    logging.info(f'Task {task_id}: Starting...')\n    for i in range(10):\n        time.sleep(sleep_duration) # Simulate work\n        worker_bar.update(1)\n        global_tqdm.update(1) # Update the global progress bar as well\n    logging.info(f'Task {task_id}: Finished.')\n    worker_bar.close()\n    return f'Task {task_id} completed'\n\nif __name__ == '__main__':\n    # Number of processes to use\n    PROCESS_COUNT = 2\n    # Total number of small steps across all tasks for the global bar\n    TOTAL_GLOBAL_STEPS = 2 * 10 # 2 tasks * 10 steps each\n\n    # Initialize a global tqdm bar (optional, can be None)\n    global_pbar = tqdm(total=TOTAL_GLOBAL_STEPS, desc='Global Progress', position=0)\n\n    # Create a list of tasks: (function_to_run, (args_tuple))\n    tasks = [\n        (worker_function, (1, 0.2)), # task_id 1, sleep 0.2s\n        (worker_function, (2, 0.3))  # task_id 2, sleep 0.3s\n    ]\n\n    # Create the TqdmMultiProcessPool\n    with TqdmMultiProcessPool(PROCESS_COUNT) as pool:\n        # Map tasks to the pool\n        results = pool.map(\n            process_count=PROCESS_COUNT,\n            global_tqdm=global_pbar,\n            task_list=tasks,\n            error_callback=lambda x: logging.error(f'Error: {x}'),\n            done_callback=lambda x: logging.info(f'Done: {x}')\n        )\n\n    global_pbar.close()\n    logging.info(f'All tasks completed. Results: {results}')","lang":"python","description":"This quickstart demonstrates how to use `TqdmMultiProcessPool` to run multiple tasks in parallel, each with its own `tqdm` progress bar, while also updating a global progress bar. It also showcases how logging from worker processes is redirected to the main process. Note the specific arguments `tqdm_func` and `global_tqdm` that must be passed to worker functions."},"warnings":[{"fix":"Always initialize worker-specific `tqdm` instances as `worker_bar = tqdm_func(total=total_steps, desc='...')` and call `worker_bar.update(1)` within your loop.","message":"Worker `tqdm` instances (those created with `tqdm_func`) do not support iterators. You must explicitly initialize them with `total=...` and update manually using `.update()`.","severity":"gotcha","affected_versions":"All versions (0.0.11)"},{"fix":"Update your `tqdm` progress bars less frequently (e.g., every few iterations) if performance issues are observed. The library author indicates a future attempt to implement a lock-free ringbuffer to improve this.","message":"Due to performance limitations of Python's default `multiprocessing.Queue`, frequent updates to global or worker `tqdm` objects can 'flood' the main process, leading to slow or non-responsive progress bars.","severity":"gotcha","affected_versions":"All versions (0.0.11)"},{"fix":"Ensure your worker function signature is `def worker_func(..., tqdm_func, global_tqdm):` and use `tqdm_func` to create any `tqdm` bars within the worker.","message":"Functions passed to `TqdmMultiProcessPool` must accept `tqdm_func` and `global_tqdm` as their last two arguments, even if `global_tqdm` is set to `None` when creating the pool.","severity":"gotcha","affected_versions":"All versions (0.0.11)"}],"env_vars":null,"search_vec":"'0.0.11':57 'aggreg':51 'allow':26 'along':44 'appear':62 'bar':18,35,49,77 'box':72 'cadenc':61 'clean':39 'current':54 'display':31 'driven':67 'easi':13 'facilit':12 'featur':66 'feature-driven':65 'global':47 'individu':33 'integr':14 'librari':10 'log':20,37,78 'main':42 'messag':38 'monitor':52 'multipl':27 'multiprocess':3,6,23,74 'parallel':80 'parallel-process':79 'process':29,43,81 'progress':17,34,48,76 'progress-bar':75 'python':9 'rather':68 'redirect':21 'releas':60 'time':71 'time-box':70 'tqdm':2,5,16,73 'tqdm-multiprocess':1,4 'version':55 'worker':28 'workflow':24","created_at":"2026-04-12T14:08:31.489815+00:00","updated_at":"2026-04-16T23:08:47.337574+00:00","problems":[{"fix":"pip install tqdm-multiprocess","cause":"The `tqdm-multiprocess` library is not installed in your current Python environment.","error":"ModuleNotFoundError: No module named 'tqdm_multiprocess'"},{"fix":"Update `tqdm` to the latest version (`pip install --upgrade tqdm`), ensure you are using `tqdm-multiprocess`'s provided wrappers (e.g., `pool_tqdm`), and restart your Jupyter kernel if necessary.","cause":"This error typically occurs in Jupyter environments due to compatibility issues or an outdated `tqdm` version conflicting with `tqdm-multiprocess`'s progress bar handling.","error":"AttributeError: 'tqdm_notebook' object has no attribute '_instances'"},{"fix":"Define any function meant to be executed by worker processes (e.g., passed to `pool_tqdm.map`) as a top-level function in a module, not as a local, nested, or lambda function.","cause":"The `multiprocessing` module (used by `tqdm-multiprocess`) requires functions and objects passed to worker processes to be picklable, which local or lambda functions often are not.","error":"_pickle.PicklingError: Can't pickle local object"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.0.11","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/EleutherAI/tqdm-multiprocess","docs":null,"changelog":null,"pypi":"https://pypi.org/project/tqdm-multiprocess/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["devops","http-networking","testing"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-30","next_check":"2026-07-28","install_tag":null}}