{"id":5215,"library":"fairscale","title":"FairScale: PyTorch Large-Scale Training Utilities","description":"FairScale is a PyTorch extension library providing utilities for large-scale and high-performance training, including Fully Sharded Data Parallel (FSDP) and Optimizer State Sharding (OSS). While many features, especially FSDP, have been upstreamed to PyTorch, FairScale offers specialized tools for memory and communication efficiency. The current version is 0.4.13. Release cadence is infrequent now, as core functionalities are integrated into PyTorch.","status":"maintenance","version":"0.4.13","language":"python","source_language":"en","source_url":"https://github.com/facebookresearch/fairscale","tags":["pytorch","distributed-training","data-parallel","fsdp","optimizer-sharding","gpu","hpc","deep-learning"],"install":[{"cmd":"pip install fairscale","lang":"bash","label":"Install FairScale"}],"dependencies":[{"reason":"Core deep learning framework. Requires torch>=1.11 for full compatibility.","package":"torch","optional":false}],"imports":[{"note":"FairScale's FSDP implementation is nested under the `data_parallel` submodule.","wrong":"from fairscale.nn import FullyShardedDataParallel","symbol":"FullyShardedDataParallel","correct":"from fairscale.nn.data_parallel import FullyShardedDataParallel"},{"symbol":"OSS","correct":"from fairscale.optim.oss import OSS"}],"quickstart":{"code":"import torch\nimport torch.nn as nn\nimport torch.distributed as dist\nfrom fairscale.nn.data_parallel import FullyShardedDataParallel as FSDP\nfrom fairscale.optim.oss import OSS\n\n# NOTE: For actual distributed use, dist.init_process_group must be called for multi-GPU/node setups.\n# This example simulates a single-process setup for quickstart.\n# In a real distributed run, rank and world_size would come from the environment.\n\n# Dummy initialization for single-process quickstart\nif not dist.is_initialized():\n    try:\n        # Using HashStore for a simple single-node, single-process initialization\n        dist.init_process_group(backend='gloo', rank=0, world_size=1, store=dist.HashStore())\n    except RuntimeError as e:\n        # Catch if already initialized (e.g., in some interactive environments)\n        print(f\"Could not initialize process group (might be already initialized): {e}\")\n\n# 1. Define a simple model\nclass MyModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.layer = nn.Linear(10, 10)\n    def forward(self, x):\n        return self.layer(x)\n\n# 2. Instantiate the model\nmodel = MyModel()\n\n# 3. Wrap the model with FairScale's FSDP\n# For simplicity, default options are used. Real-world usage often requires careful tuning.\nfsdp_model = FSDP(model)\n\n# 4. Wrap the optimizer with FairScale's OSS\noptimizer = torch.optim.Adam(fsdp_model.parameters(), lr=1e-3)\noss_optimizer = OSS(params=fsdp_model.parameters(), optim=optimizer)\n\n# 5. Dummy data and training step\ninput_data = torch.randn(2, 10)\nlabels = torch.randn(2, 10)\n\n# Forward pass\noutput = fsdp_model(input_data)\nloss = nn.MSELoss()(output, labels)\n\n# Backward pass and optimizer step\noss_optimizer.zero_grad()\nloss.backward()\noss_optimizer.step()\n\nprint(f\"FairScale FSDP and OSS example completed. Loss: {loss.item():.4f}\")\n\n# Clean up distributed environment if it was initialized by this script\nif dist.is_initialized() and dist.get_world_size() == 1:\n    dist.destroy_process_group()","lang":"python","description":"This quickstart demonstrates how to wrap a PyTorch model with FairScale's Fully Sharded Data Parallel (FSDP) and its Optimizer State Sharding (OSS) for memory-efficient training. Note that `dist.init_process_group` is essential for multi-GPU/node training; a dummy initialization is used here for a runnable single-process example. For new projects, it is highly recommended to consider migrating to PyTorch's native FSDP."},"warnings":[{"fix":"Migrate your FSDP usage to `torch.distributed.fsdp.FullyShardedDataParallel`. Consult the official PyTorch FSDP documentation for migration guides and updated best practices.","message":"FairScale's FSDP (`fairscale.nn.data_parallel.FullyShardedDataParallel`) is largely superseded by PyTorch's native FSDP (`torch.distributed.fsdp.FullyShardedDataParallel`) since PyTorch 1.11 and 1.12+. For new projects, the native PyTorch implementation is strongly encouraged due to ongoing development and optimizations.","severity":"deprecated","affected_versions":"0.4.0+"},{"fix":"Plan for migration to native PyTorch distributed features, especially `torch.distributed.fsdp`, to ensure future compatibility, access to the latest optimizations, and bug fixes.","message":"FairScale is in maintenance mode, meaning active development for new features has largely shifted to PyTorch's native distributed modules. Future API changes or new features in PyTorch's core distributed components might not be backported or fully compatible with FairScale in the future.","severity":"breaking","affected_versions":"0.4.0+"},{"fix":"Ensure `torch.distributed.init_process_group` is called before instantiating FairScale's FSDP or OSS. Use environment variables (e.g., `MASTER_ADDR`, `MASTER_PORT`, `RANK`, `WORLD_SIZE`) or helper functions for distributed setup.","message":"FairScale requires a properly initialized `torch.distributed` environment. Running without `dist.init_process_group` (even for single-GPU FSDP) will result in errors or unexpected behavior during model wrapping or training.","severity":"gotcha","affected_versions":"All"},{"fix":"Refer to FairScale's documentation on mixed precision usage with FSDP. In many cases, `torch.cuda.amp` can be used alongside FSDP, but careful integration is required.","message":"When using FairScale's FSDP with mixed precision, ensure that the `mixed_precision` argument in `FSDP` is configured correctly, or that you are using a compatible `torch.cuda.amp.GradScaler` outside of FSDP, depending on your PyTorch version and specific setup. Incorrect configuration can lead to performance issues or `NaN` gradients.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"search_vec":"'0.4.13':59 'cadenc':61 'communic':53 'core':66 'current':56 'data':28,77 'data-parallel':76 'deep':86 'deep-learn':85 'distribut':74 'distributed-train':73 'effici':54 'especi':39 'extens':12 'fairscal':1,8,46 'featur':38 'fsdp':30,40,79 'fulli':26 'function':67 'gpu':83 'high':22 'high-perform':21 'hpc':84 'includ':25 'infrequ':63 'integr':69 'larg':4,18 'large-scal':3,17 'learn':87 'librari':13 'mani':37 'memori':51 'offer':47 'optim':32,81 'optimizer-shard':80 'oss':35 'parallel':29,78 'perform':23 'provid':14 'pytorch':2,11,45,71,72 'releas':60 'scale':5,19 'shard':27,34,82 'special':48 'state':33 'tool':49 'train':6,24,75 'upstream':43 'util':7,15 'version':57","created_at":"2026-04-14T01:24:11.210089+00:00","updated_at":"2026-04-16T14:54:54.913088+00:00","problems":[{"fix":"Install FairScale using pip: `pip install fairscale` or `pip install fairscale==0.4.13` for the specific version. Ensure your virtual environment is activated if applicable.","cause":"The FairScale library is not installed in your current Python environment, or the environment where it was installed is not active.","error":"ModuleNotFoundError: No module named 'fairscale'"},{"fix":"For new projects, use `torch.distributed.fsdp.FullyShardedDataParallel`. For existing projects, plan for migration to native PyTorch FSDP to ensure future compatibility and access to latest optimizations.","cause":"FairScale's FSDP (`fairscale.nn.data_parallel.FullyShardedDataParallel`) has largely been superseded by PyTorch's native FSDP (`torch.distributed.fsdp.FullyShardedDataParallel`) since PyTorch 1.11 and 1.12+. FairScale is in maintenance mode, with active development shifting to PyTorch's native distributed modules.","error":"fairscale FSDP deprecated"},{"fix":"Ensure `torch.distributed.init_process_group` is called early in your distributed training setup, providing necessary parameters like `backend`, `init_method`, `rank`, and `world_size`.","cause":"FairScale, especially FSDP, requires a properly initialized `torch.distributed` environment. This error occurs if `torch.distributed.init_process_group` has not been called before instantiating FairScale's FSDP or OSS modules.","error":"RuntimeError: Default process group has not been initialized"},{"fix":"Review your FSDP wrapping strategy, especially with nested modules or activation checkpointing. Ensure compatibility between your PyTorch and FairScale versions. Consider updating FairScale or adapting to PyTorch's native FSDP which might resolve underlying parameter management issues.","cause":"This error typically arises from internal inconsistencies or incompatible usage patterns with `FullyShardedDataParallel`, possibly related to how parameters are flattened and managed internally, or specific versions of PyTorch/FairScale.","error":"AttributeError: 'FlatParameter' object has no attribute '_full_param_padded'"},{"fix":"Consider reducing batch size, enabling mixed precision training (e.g., `mixed_precision=True` in FSDP, `torch.cuda.amp.autocast`), using CPU offloading (`cpu_offload=True`), implementing activation checkpointing, or strategically wrapping layers in FSDP to optimize memory usage.","cause":"Despite using FSDP for memory efficiency, large models or specific training configurations (e.g., high batch size, long sequences, lack of mixed precision) can still lead to GPU Out Of Memory (OOM) errors.","error":"Out Of Memory Error (FairScale FSDP)"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.4.13","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":null,"docs":null,"changelog":null,"pypi":"https://pypi.org/project/fairscale/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["ai-ml"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-30","next_check":"2026-07-28","install_tag":null}}