{"id":6022,"library":"pandarallel","title":"Pandarallel","description":"Pandarallel is a Python library that extends Pandas to support parallel processing across multiple CPU cores. It aims to significantly speed up Pandas operations on large datasets by distributing computations, often requiring only a one-line code change. The library also provides progress bars. It is currently at version 1.6.5 and is actively maintained.","status":"active","version":"1.6.5","language":"python","source_language":"en","source_url":"https://github.com/nalepae/pandarallel","tags":["pandas","parallel-processing","performance","dataframe","multiprocessing"],"install":[{"cmd":"pip install pandarallel","lang":"bash","label":"Install with pip"}],"dependencies":[{"reason":"Pandarallel is built on top of Pandas and parallelizes its operations.","package":"pandas","optional":false},{"reason":"Frequently used in conjunction with Pandas for numerical operations.","package":"numpy","optional":false}],"imports":[{"symbol":"pandarallel","correct":"from pandarallel import pandarallel"}],"quickstart":{"code":"import pandas as pd\nfrom pandarallel import pandarallel\nimport os\n\n# Initialize pandarallel. It defaults to using all available CPU cores.\n# progress_bar=True is often useful to visualize progress.\npandarallel.initialize(nb_workers=os.cpu_count(), progress_bar=True)\n\n# Create a sample DataFrame with some data\ndata = {'col1': range(1_000_000), 'col2': [f'item_{i}' for i in range(1_000_000)]}\ndf = pd.DataFrame(data)\n\n# Define a CPU-bound function to apply\ndef example_computation(x):\n    # Simulate a computationally intensive task\n    res = 0\n    for i in range(50):\n        res += (x * i) ** 0.5\n    return res\n\n# Apply the function in parallel using pandarallel's parallel_apply\n# This replaces df['col1'].apply(example_computation)\nprint(\"Starting parallel computation...\")\ndf['result'] = df['col1'].parallel_apply(example_computation)\n\nprint(\"Computation complete. First 5 rows of the DataFrame with results:\")\nprint(df.head())","lang":"python","description":"This quickstart demonstrates how to initialize pandarallel and then use `parallel_apply` on a Pandas Series. It includes a simple, CPU-bound function to showcase the parallelization effect. The `nb_workers` is explicitly set to the CPU count for clarity, and a progress bar is enabled."},"warnings":[{"fix":"Monitor memory usage. For data larger than available memory, consider alternatives like Dask or PySpark.","message":"Pandarallel can require up to twice the memory of standard Pandas operations. Ensure your system has sufficient RAM, especially for large datasets.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Define functions at the top level of the module. For complex scenarios, consider using Windows Subsystem for Linux (WSL) or refactor functions to be entirely self-contained.","message":"On Windows, functions passed to `pandarallel` must be self-contained and should not depend on external resources (e.g., global variables, complex closures) due to Python's `multiprocessing` 'spawn' start method.","severity":"gotcha","affected_versions":"All versions on Windows"},{"fix":"Benchmark performance with and without `pandarallel` for your specific use case to determine if parallelization is beneficial.","message":"Parallelization introduces overhead. For small datasets or very fast operations, `pandarallel` might not provide a speedup, or could even be slower than native Pandas.","severity":"gotcha","affected_versions":"All versions"},{"fix":"For optimal performance, set `nb_workers` to your system's number of physical CPU cores or allow `pandarallel` to determine it automatically.","message":"Pandarallel scales best with the number of *physical* CPU cores, not necessarily logical cores (hyperthreading). Setting `nb_workers` higher than physical cores may not yield further performance gains.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Remove `shm_size_mb` from `pandarallel.initialize()` calls. Memory file system usage is now controlled by `use_memory_fs`.","message":"The `shm_size_mb` parameter in `pandarallel.initialize()` is deprecated and should no longer be used.","severity":"gotcha","affected_versions":">=1.x.x"},{"fix":"Monitor system CPU usage. Try reducing `nb_workers` to leave some cores free, or ensure your environment has sufficient idle CPU resources.","message":"Pandarallel can sometimes get stuck without raising errors if all physical cores are heavily utilized by other background processes. The progress bar might stop updating.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure functions passed to `pandarallel` methods are defined at the top level of a module, not nested within other functions.","message":"Functions defined locally (e.g., inside another function) or using closures may lead to `AttributeError: Can't pickle local object` errors, a common issue with Python's multiprocessing.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"search_vec":"'1.6.5':52 'across':14 'activ':55 'aim':19 'also':43 'bar':46 'chang':40 'code':39 'comput':31 'core':17 'cpu':16 'current':49 'datafram':62 'dataset':28 'distribut':30 'extend':8 'larg':27 'librari':6,42 'line':38 'maintain':56 'multipl':15 'multiprocess':63 'often':32 'one':37 'one-lin':36 'oper':25 'panda':9,24,57 'pandarallel':1,2 'parallel':12,59 'parallel-process':58 'perform':61 'process':13,60 'progress':45 'provid':44 'python':5 'requir':33 'signific':21 'speed':22 'support':11 'version':51","created_at":"2026-04-14T18:38:34.289453+00:00","updated_at":"2026-04-17T14:55:21.631924+00:00","problems":[{"fix":"pip install pandarallel","cause":"The 'pandarallel' library has not been installed in the current Python environment.","error":"ModuleNotFoundError: No module named 'pandarallel'"},{"fix":"Define the function at the top level of the module (globally) or ensure it is a static method if part of a class, so it can be properly pickled and accessed by worker processes.","cause":"Functions passed to `pandarallel` methods (which use Python's multiprocessing) must be picklable. This error occurs when a function is defined locally (e.g., inside another function or a method of a class) and thus cannot be serialized for distribution to worker processes. This is especially common on Windows due to its default 'spawn' multiprocessing start method.","error":"AttributeError: Can't pickle local object 'prepare_worker..closure..wrapper'"},{"fix":"Ensure `pandarallel.initialize()` is called once at the beginning of your script. For `groupby().apply()`, use `df.groupby(...).parallel_apply(func)`. Note that `pandarallel` only supports specific parallelized Pandas APIs.","cause":"This usually means `pandarallel` has not been initialized with `pandarallel.initialize()` before attempting to use its parallelized methods, or the method is called on an unsupported Pandas object/operation.","error":"AttributeError: 'DataFrameGroupBy' object has no attribute 'parallel_apply'"},{"fix":"Refactor the function to be entirely self-contained, ensuring it does not rely on global variables or objects defined outside its scope. Define helper functions at the top level of the module. For complex scenarios, consider using Windows Subsystem for Linux (WSL).","cause":"On Windows, due to the `multiprocessing` 'spawn' start method, functions passed to `pandarallel` must be self-contained and cannot depend on external resources (like global variables or complex closures) defined in the main script.","error":"pandarallel does not work at all. On Windows, because of the multiprocessing system (spawn), the function you send to pandarallel must be self contained."},{"fix":"Call `pandarallel.initialize()` after importing `pandarallel` to enable the parallel methods:\n```python\nimport pandas as pd\nfrom pandarallel import pandarallel\n\npandarallel.initialize()\n\ndf = pd.DataFrame({'a': range(100)})\ndf['b'] = df.parallel_apply(lambda x: x['a'] * 2, axis=1)\n```","cause":"The `pandarallel` library was imported but its `initialize()` method was not called, preventing it from patching Pandas DataFrames and Series with parallel methods.","error":"AttributeError: 'DataFrame' object has no attribute 'parallel_apply'"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"1.6.5","cli_name":"","cli_version":null,"type":"library","homepage":"https://nalepae.github.io/pandarallel","github":null,"docs":null,"changelog":null,"pypi":"https://pypi.org/project/pandarallel/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","workflow"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-06-28","next_check":"2026-07-28","install_tag":null}}