{"id":8364,"library":"nvidia-resiliency-ext","title":"NVIDIA Resiliency Extension","description":"NVIDIA Resiliency Extension (NVRE) is a Python package that provides fault-tolerant features for framework developers and users, aiming to minimize downtime in deep learning training due to failures and interruptions. It supports features like checkpointing (local and cloud), in-job restarts, and health checks. The current version is 0.5.0, with minor releases occurring every few months to introduce new features and bug fixes.","status":"active","version":"0.5.0","language":"python","source_language":"en","source_url":"https://github.com/NVIDIA/nvidia-resiliency-ext","tags":["NVIDIA","GPU","deep-learning","fault-tolerance","resilience","checkpointing","distributed-training","MLOps"],"install":[{"cmd":"pip install nvidia-resiliency-ext","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Required for core functionalities.","package":"numpy","optional":false},{"reason":"Required for MPI-based resiliency features and distributed training contexts.","package":"mpi4py","optional":true},{"reason":"Often used for data serialization, especially in distributed contexts.","package":"protobuf","optional":true},{"reason":"Required for S3 cloud checkpointing support.","package":"boto3","optional":true},{"reason":"Required for Google Cloud Storage checkpointing support.","package":"google-cloud-storage","optional":true}],"imports":[{"wrong":"import nvre","symbol":"nvidia_resiliency_ext","correct":"import nvidia_resiliency_ext"}],"quickstart":{"code":"import nvre\nimport os\n\ndef my_restart_callback(restart_args):\n    print(f\"[NVRE] Restart event received: {restart_args}\")\n\n# 1. Initialize the resiliency manager\n# This must be called early in your application's lifecycle.\n# In a real scenario, this might be within a distributed setup like Horovod or PyTorch DDP.\n# For simple testing, it can run standalone.\nprint(\"[NVRE] Initializing resiliency manager...\")\nnvre.init_resiliency_manager()\n\n# 2. Register a callback for restart events (optional, but good practice)\nnvre.register_restart_callback(my_restart_callback)\n\n# 3. Example: Checkpointing\ncheckpoint_id = \"my_training_state\"\n\nif nvre.has_checkpoint(checkpoint_id):\n    print(f\"[NVRE] Loading checkpoint '{checkpoint_id}'...\")\n    state = nvre.load_checkpoint(checkpoint_id)\n    current_step = state.get(\"step\", 0)\n    print(f\"[NVRE] Resuming from step {current_step}\")\nelse:\n    print(f\"[NVRE] No checkpoint found for '{checkpoint_id}'. Starting new training.\")\n    current_step = 0\n\n# Simulate some training steps\nfor i in range(current_step, current_step + 3):\n    print(f\"[NVRE] Training step {i}\")\n    # Simulate a checkpoint save every step for demonstration\n    if i % 1 == 0:\n        state_to_save = {\"step\": i + 1, \"model_config\": {\"lr\": 0.001}}\n        print(f\"[NVRE] Saving checkpoint '{checkpoint_id}' at step {i}...\")\n        nvre.save_checkpoint(checkpoint_id, state_to_save)\n\nprint(\"[NVRE] Training finished.\")\n# Cleanup (optional in many cases, but good for explicit shutdown)\nnvre.shutdown_resiliency_manager()","lang":"python","description":"This quickstart demonstrates the basic initialization of the resiliency manager, registering a restart callback, and using checkpointing to save and load training state. The example simulates training steps and checkpoint saves, showing how to resume from a previously saved state."},"warnings":[{"fix":"Ensure `nvre.init_resiliency_manager()` is the first NVRE call in your main execution path.","message":"The `nvre.init_resiliency_manager()` function must be called early in your application's execution before using any other NVRE features. Failing to do so will result in `RuntimeError`.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Install necessary optional dependencies for your chosen resiliency backend, e.g., `pip install mpi4py` or `pip install boto3` for cloud checkpointing.","message":"For distributed training or specific resiliency backends (e.g., MPI), additional dependencies like `mpi4py` might be required but are not direct dependencies of the `nvidia-resiliency-ext` package. Using these features without the corresponding packages installed will lead to import errors or runtime failures.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Install the appropriate SDK for your cloud provider: `pip install boto3` for AWS S3, `pip install google-cloud-storage` for GCS.","message":"Cloud checkpointing features (e.g., S3, GCS) introduced in v0.4.0 require installing specific cloud provider SDKs (e.g., `boto3` for AWS S3, `google-cloud-storage` for GCS). Without these, attempts to use cloud storage will fail.","severity":"gotcha","affected_versions":">=0.4.0"}],"env_vars":null,"search_vec":"'0.5.0':55 'aim':23 'bug':68 'check':50 'checkpoint':40,79 'cloud':43 'current':52 'deep':28,73 'deep-learn':72 'develop':20 'distribut':81 'distributed-train':80 'downtim':26 'due':31 'everi':60 'extens':3,6 'failur':33 'fault':15,76 'fault-toler':14,75 'featur':17,38,66 'fix':69 'framework':19 'gpu':71 'health':49 'in-job':44 'interrupt':35 'introduc':64 'job':46 'learn':29,74 'like':39 'local':41 'minim':25 'minor':57 'mlop':83 'month':62 'new':65 'nvidia':1,4,70 'nvre':7 'occur':59 'packag':11 'provid':13 'python':10 'releas':58 'resili':2,5,78 'restart':47 'support':37 'toler':16,77 'train':30,82 'user':22 'version':53","created_at":"2026-04-16T17:01:48.590779+00:00","updated_at":"2026-04-16T17:01:48.590779+00:00","problems":null,"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.6.0","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/NVIDIA/nvidia-resiliency-ext","docs":null,"changelog":null,"pypi":"https://pypi.org/project/nvidia-resiliency-ext/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["ai-ml","data"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-30","last_verified":"2026-06-30","next_check":"2026-07-30","install_tag":null}}