{"id":669,"library":"onnxruntime","title":"ONNX Runtime","description":"ONNX Runtime is a cross-platform, high-performance machine learning inference and training accelerator. It enables faster customer experiences and lower costs by supporting models from various deep learning frameworks (e.g., PyTorch, TensorFlow/Keras) and classical ML libraries (e.g., scikit-learn). The library is actively maintained with new releases approximately quarterly, including patch releases, and commits to backwards compatibility.","status":"active","version":"1.24.4","language":"python","source_language":"en","source_url":"https://github.com/microsoft/onnxruntime","tags":["machine-learning","inference","onnx","deep-learning","runtime","gpu-acceleration","cpu-optimization","model-deployment"],"install":[{"cmd":"pip install onnxruntime","lang":"bash","label":"CPU Version (default)"},{"cmd":"pip install onnxruntime-gpu","lang":"bash","label":"GPU Version (CUDA 12.x)"},{"cmd":"pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-11/pypi/simple/","lang":"bash","label":"GPU Version (CUDA 11.8)"}],"dependencies":[{"reason":"Essential for numerical data handling, especially for model inputs/outputs.","package":"numpy","optional":false},{"reason":"Required for loading, checking, and working with ONNX model files.","package":"onnx","optional":true},{"reason":"Provides custom operators for advanced pre- and post-processing, requiring separate installation and registration.","package":"onnxruntime-extensions","optional":true},{"reason":"Necessary system-level dependencies for `onnxruntime-gpu` to leverage NVIDIA GPUs. Versions must be compatible with the installed `onnxruntime-gpu` package.","package":"CUDA Toolkit / cuDNN","optional":true}],"imports":[{"symbol":"InferenceSession","correct":"from onnxruntime import InferenceSession"},{"symbol":"SessionOptions","correct":"from onnxruntime import SessionOptions"},{"symbol":"get_available_providers","correct":"from onnxruntime import get_available_providers"},{"note":"For using custom operators provided by `onnxruntime-extensions`.","symbol":"PyOrtFunction","correct":"from onnxruntime_extensions import PyOrtFunction"}],"quickstart":{"code":"import onnxruntime as ort\nimport numpy as np\nimport os\n\n# NOTE: This example assumes you have an ONNX model file named 'model.onnx'.\n# You can typically export models from frameworks like PyTorch or TensorFlow to ONNX format.\n# For a runnable example, you'd need to create a dummy model:\n# e.g., using ONNX library:\n# import onnx\n# from onnx import TensorProto\n# from onnx.helper import make_model, make_node, make_graph, make_tensor_value_info\n# X = make_tensor_value_info('input', TensorProto.FLOAT, [None, 2])\n# Y = make_tensor_value_info('output', TensorProto.FLOAT, [None, 2])\n# node = make_node('Add', ['input', 'input'], ['output'])\n# graph = make_graph([node], 'simple-graph', [X], [Y])\n# onnx_model = make_model(graph)\n# onnx.save(onnx_model, 'model.onnx')\n\nmodel_path = os.environ.get('ONNX_MODEL_PATH', 'model.onnx')\n\n# 1. Create an InferenceSession\n# For GPU, add providers=['CUDAExecutionProvider', 'CPUExecutionProvider']\nsession = ort.InferenceSession(model_path, providers=ort.get_available_providers())\n\nprint(\"Model inputs:\")\nfor input_meta in session.get_inputs():\n    print(f\"  Name: {input_meta.name}, Shape: {input_meta.shape}, Type: {input_meta.type}\")\n\nprint(\"\\nModel outputs:\")\nfor output_meta in session.get_outputs():\n    print(f\"  Name: {output_meta.name}, Shape: {output_meta.shape}, Type: {output_meta.type}\")\n\n# 2. Prepare input data (example for a model expecting a float32 array)\n# Assuming the first input expects a 2D float32 array, e.g., shape (1, 2)\ninput_name = session.get_inputs()[0].name\ninput_shape = [dim if isinstance(dim, int) else 1 for dim in session.get_inputs()[0].shape] # Handle dynamic shapes\ninput_data = np.random.randn(*input_shape).astype(np.float32)\n\n# 3. Run inference\noutputs = session.run(None, {input_name: input_data})\n\n# 4. Process outputs\nprint(f\"\\nOutput data type: {outputs[0].dtype}\")\nprint(f\"Output shape: {outputs[0].shape}\")\nprint(f\"Output data (first 5 elements): {outputs[0].flatten()[:5]}\")\n","lang":"python","description":"This quickstart demonstrates how to load an ONNX model, inspect its inputs and outputs, prepare sample input data using NumPy, and perform inference using `onnxruntime.InferenceSession`. It also shows how to configure execution providers for CPU or GPU inference. A `model.onnx` file is required for this code to run; a comment in the code suggests how to create a simple dummy model using the `onnx` library."},"warnings":[{"fix":"Uninstall any conflicting packages (`pip uninstall onnxruntime onnxruntime-gpu`) before installing the desired version.","message":"The `onnxruntime` and `onnxruntime-gpu` packages are mutually exclusive. Only one should be installed in a given Python environment. Installing both can lead to unexpected behavior or errors.","severity":"breaking","affected_versions":"All versions"},{"fix":"Pass a list of providers to `ort.InferenceSession(model_path, providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])`. Use `ort.get_available_providers()` to see what's available.","message":"Since ONNX Runtime 1.10, execution providers (like CUDAExecutionProvider for GPU) must be explicitly specified when creating an `InferenceSession`. If not specified, it defaults to `CPUExecutionProvider` only. Older code that relied on implicit GPU usage will break or silently fall back to CPU.","severity":"breaking","affected_versions":">=1.10.0"},{"fix":"Inspect `session.get_inputs()` to verify expected `shape` and `type`. Ensure NumPy arrays are created with the correct `dtype` (e.g., `np.float32`). Reshape inputs using `np.reshape` or `np.expand_dims` as needed.","message":"Input data shapes and data types must precisely match the ONNX model's expected inputs, otherwise `onnxruntime` will raise `INVALID_ARGUMENT` errors. Common mistakes include incorrect dimensions or using `float64` instead of the expected `float32`.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Consult the ONNX Runtime documentation for the exact CUDA/cuDNN version requirements for your `onnxruntime-gpu` package. Ensure they are installed and discoverable in your system's PATH/LD_LIBRARY_PATH.","message":"When using `onnxruntime-gpu`, correct installation of the CUDA Toolkit and cuDNN libraries matching your `onnxruntime-gpu` version is crucial. Mismatched versions are a frequent cause of 'DLL not found' or 'Failed to create session' errors.","severity":"gotcha","affected_versions":"All `onnxruntime-gpu` versions"},{"fix":"Update `onnxruntime-genai` code to use the new `append_tokens` method and remove `compute_logits` calls, as detailed in the migration guide.","message":"The `generate()` API for generative AI models underwent breaking changes from ONNX Runtime GenAI 0.5.2 to 0.6.0, notably replacing `params.input_ids = input_tokens` with `generator.append_tokens(input_tokens)` and removing `generator.compute_logits()`.","severity":"deprecated","affected_versions":"0.5.x -> 0.6.x for `onnxruntime-genai`"},{"fix":"Install `pip install onnxruntime-extensions` and use `so = ort.SessionOptions(); so.register_custom_ops_library(get_library_path()); sess = ort.InferenceSession(model, sess_options=so)` where `get_library_path` comes from `onnxruntime_extensions`.","message":"For pre- and post-processing steps using custom ONNX operators, the `onnxruntime-extensions` package must be separately installed and its custom operators registered with the `InferenceSession` via `session_options.register_custom_ops_library()`.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Use a supported environment: try a stable Python version (e.g., 3.8-3.11) and a common Linux distribution (e.g., Debian/Ubuntu based with glibc). Check the official ONNX Runtime documentation for supported environments and available wheels.","message":"`onnxruntime` relies heavily on pre-built wheels. `pip` may fail to find a matching distribution if wheels are not available for your specific Python version (e.g., Python 3.13+) or operating system/architecture (e.g., Alpine Linux which uses musl libc, or ARM64 where fewer wheels are available). Trying to build from source can be complex and often fails.","severity":"breaking","affected_versions":"All `onnxruntime` versions"}],"env_vars":null,"search_vec":"'acceler':18,75 'activ':49 'approxim':54 'backward':62 'classic':39 'commit':60 'compat':63 'cost':26 'cpu':77 'cpu-optim':76 'cross':8 'cross-platform':7 'custom':22 'deep':32,70 'deep-learn':69 'deploy':81 'e.g':35,42 'enabl':20 'experi':23 'faster':21 'framework':34 'gpu':74 'gpu-acceler':73 'high':11 'high-perform':10 'includ':56 'infer':15,67 'learn':14,33,45,66,71 'librari':41,47 'lower':25 'machin':13,65 'machine-learn':64 'maintain':50 'ml':40 'model':29,80 'model-deploy':79 'new':52 'onnx':1,3,68 'optim':78 'patch':57 'perform':12 'platform':9 'pytorch':36 'quarter':55 'releas':53,58 'runtim':2,4,72 'scikit':44 'scikit-learn':43 'support':28 'tensorflow/keras':37 'train':17 'various':31","created_at":"2026-03-28T17:09:44.272969+00:00","updated_at":"2026-04-17T15:04:51.883034+00:00","problems":[{"fix":"Install the module using pip: 'pip install onnxruntime'.","cause":"The 'onnxruntime' module is not installed in the Python environment.","error":"ModuleNotFoundError: No module named 'onnxruntime'"},{"fix":"Ensure 'onnxruntime' is installed correctly and rename any script named 'onnxruntime.py' to avoid conflicts.","cause":"The 'onnxruntime' module is either not installed correctly or the script's filename conflicts with the module name.","error":"AttributeError: module 'onnxruntime' has no attribute 'InferenceSession'"},{"fix":"Upgrade 'onnxruntime' to the latest version using pip: 'pip install --upgrade onnxruntime'.","cause":"The 'OrtValue' attribute is not available in the installed version of 'onnxruntime'.","error":"AttributeError: module 'onnxruntime' has no attribute 'OrtValue'"},{"fix":"Upgrade 'onnxruntime' to the latest version using pip: 'pip install --upgrade onnxruntime'.","cause":"The 'SessionOptions' attribute is not available in the installed version of 'onnxruntime'.","error":"AttributeError: module 'onnxruntime' has no attribute 'SessionOptions'"},{"fix":"Convert your input data to the expected data type, typically `numpy.float32`, before feeding it to the ONNX Runtime session, e.g., `input_data.astype(numpy.float32)`.","cause":"The input data provided to the ONNX Runtime session has a data type (e.g., `numpy.float64`) that does not match the expected data type of the ONNX model (e.g., `numpy.float32`).","error":"[ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(double)) , expected: (tensor(float))."}],"ecosystem":"pypi","meta_description":null,"install_score":50,"quickstart_score":0,"quickstart_tag":"stale","pypi_latest":"1.26.0","cli_name":"","cli_version":null,"type":"library","homepage":"https://onnxruntime.ai","github":"https://github.com/microsoft/onnxruntime","docs":null,"changelog":null,"pypi":"https://pypi.org/project/onnxruntime/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["ai-ml","data"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-06-28","next_check":"2026-07-28","install_tag":"draft"}}