{"id":2315,"library":"torchao","title":"PyTorch AO (torchao)","description":"TorchAO is a PyTorch library for applying advanced optimization (AO) techniques, primarily quantization and sparsity, to deep learning models running on GPUs. It focuses on performance acceleration through low-precision kernels, mixture-of-experts (MoE) optimizations, and quantization-aware training (QAT). The current version is 0.17.0, with new versions and significant features released frequently, often monthly.","status":"active","version":"0.17.0","language":"python","source_language":"en","source_url":"https://github.com/pytorch/ao","tags":["pytorch","quantization","optimization","deep-learning","gpu","low-precision"],"install":[{"cmd":"pip install torchao","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"Core PyTorch dependency for model definition and operations.","package":"torch","optional":false}],"imports":[{"wrong":"from torchao.quantization import quantize_","symbol":"quantize_","correct":"from torchao.quantization import quantize_"}],"quickstart":{"code":"import torch\nimport torch.nn as nn\nfrom torchao.quantization import quantize_, int8_dynamic_activation_int4_weight\n\n# 1. Define a simple model\nclass MyModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.linear1 = nn.Linear(10, 20)\n        self.relu = nn.ReLU()\n        self.linear2 = nn.Linear(20, 5)\n\n    def forward(self, x):\n        return self.linear2(self.relu(self.linear1(x)))\n\nmodel = MyModel()\nprint(f\"Original model: {model}\")\n\n# 2. Define a quantization recipe\n# This uses a predefined post-training quantization recipe\nquantizer = int8_dynamic_activation_int4_weight()\n\n# 3. Apply quantization to the model\n# quantize_ modifies the model in-place (or returns a modified copy)\nquantized_model = quantize_(model, quantizer)\n\nprint(f\"\\nQuantized model: {quantized_model}\")\n\n# Test with some dummy input\ndummy_input = torch.randn(1, 10)\noutput = quantized_model(dummy_input)\nprint(f\"\\nOutput shape: {output.shape}\")\nassert isinstance(quantized_model.linear1, torch.nn.Module) # Verify structure","lang":"python","description":"This quickstart demonstrates how to define a simple PyTorch model and apply a predefined post-training quantization recipe using `torchao.quantization.quantize_`."},"warnings":[{"fix":"Consult the `torchao` documentation for versions 0.9.0+ for the new `quantize_` API, which now typically requires a `Quantizer` object (e.g., from a predefined recipe or a custom `QuantConfig`).","message":"The `quantize_` API underwent a significant overhaul in version 0.9.0, changing how quantization recipes are applied to models. Direct calls to `quantize_` with previous argument patterns will fail.","severity":"breaking","affected_versions":"<0.9.0"},{"fix":"Refer to the latest `torchao` documentation (v0.16.0+) for current recommended configurations and quantization options. Update your code to use the officially supported APIs.","message":"Older configurations and less-used quantization options have been deprecated to streamline the library. Using these deprecated features may lead to warnings or errors in future releases.","severity":"deprecated","affected_versions":"<0.16.0"},{"fix":"Avoid using `torchao.prototype` modules in production-critical code. If used, be prepared for frequent updates and potential breaking changes with new `torchao` releases.","message":"Features located in `torchao.prototype` modules are experimental and subject to frequent, unannounced API changes, or may be removed entirely without prior deprecation. They are not considered stable.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Before deployment, verify your CUDA version and GPU architecture against `torchao`'s requirements for the specific features you intend to use. Ensure your environment matches the recommendations for optimal results.","message":"Optimal performance for `torchao`'s advanced kernels (e.g., MXFP8 MoE, W4A8) often requires specific CUDA versions (e.g., CUDA 12.8+) or particular GPU architectures (e.g., Blackwell, GB200). Using non-supported environments may result in reduced performance, errors, or inability to leverage certain features.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"search_vec":"'0.17.0':52 'acceler':30 'advanc':11 'ao':2,13 'appli':10 'awar':45 'current':49 'deep':20,67 'deep-learn':66 'expert':39 'featur':58 'focus':27 'frequent':60 'gpu':69 'gpus':25 'kernel':35 'learn':21,68 'librari':8 'low':33,71 'low-precis':32,70 'mixtur':37 'mixture-of-expert':36 'model':22 'moe':40 'month':62 'new':54 'often':61 'optim':12,41,65 'perform':29 'precis':34,72 'primarili':15 'pytorch':1,7,63 'qat':47 'quantiz':16,44,64 'quantization-awar':43 'releas':59 'run':23 'signific':57 'sparsiti':18 'techniqu':14 'torchao':3,4 'train':46 'version':50,55","created_at":"2026-04-09T18:52:43.873997+00:00","updated_at":"2026-04-09T18:52:43.873997+00:00","problems":{"verify_error":"Traceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"/tmp/tmpn318d8jl/venv/lib/python3.12/site-packages/torchao/__init__.py\", line 8, in <module>\n    import torch\nModuleNotFoundError: No module named 'torch'"},"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.18.0","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/pytorch/ao","docs":null,"changelog":null,"pypi":"https://pypi.org/project/torchao/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["ai-ml","llm-agents"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-07-03","last_verified":"2026-08-28","next_check":"2026-07-10","install_tag":null}}