{"id":4124,"library":"mteb","title":"Massive Text Embedding Benchmark (MTEB)","description":"MTEB (Massive Text Embedding Benchmark) is a Python framework for evaluating embeddings and retrieval systems across diverse NLP tasks, including classification, clustering, retrieval, reranking, and semantic textual similarity. It supports over 1000 languages and various modalities like text and image, with continuous expansion. As of version 2.12.16, it aims to provide a standardized, comprehensive, and reproducible way to compare embedding models. The library maintains a frequent release cadence with minor updates often occurring weekly.","status":"active","version":"2.12.16","language":"python","source_language":"en","source_url":"https://github.com/embeddings-benchmark/mteb","tags":["embeddings","benchmarking","nlp","evaluation","sentence-transformers","huggingface","multimodal","retrieval","classification","clustering","semantic-textual-similarity"],"install":[{"cmd":"pip install mteb","lang":"bash","label":"Install core library"},{"cmd":"uv add mteb","lang":"bash","label":"Faster installation with uv"}],"dependencies":[{"reason":"Runtime environment","package":"python","version":">=3.10, <3.15","optional":false},{"reason":"Commonly used for loading and evaluating many pre-trained models. MTEB also offers its own model loading mechanism.","package":"sentence-transformers","optional":false},{"reason":"Underlying deep learning framework, implicitly required by sentence-transformers and many models.","package":"torch","optional":false},{"reason":"Underlying library for many models and tokenizers.","package":"transformers","optional":false}],"imports":[{"note":"The MTEB class was part of a major refactor in v2; direct use is less common than 'mteb.evaluate' or 'mteb.get_model' now.","wrong":"from mteb.MTEB import MTEB","symbol":"MTEB","correct":"from mteb import MTEB"},{"note":"The recommended way to run evaluations.","symbol":"evaluate","correct":"import mteb\nresults = mteb.evaluate(model, tasks=tasks)"},{"note":"Recommended for loading existing models as implemented in MTEB for reproducibility.","symbol":"get_model","correct":"import mteb\nmodel = mteb.get_model('sentence-transformers/all-MiniLM-L6-v2')"},{"note":"Used to select specific benchmark tasks.","symbol":"get_tasks","correct":"import mteb\ntasks = mteb.get_tasks(tasks=['Banking77Classification.v2'])"},{"note":"Used to load models that are not yet directly implemented in MTEB's registry.","symbol":"SentenceTransformer","correct":"from sentence_transformers import SentenceTransformer"}],"quickstart":{"code":"import mteb\nfrom sentence_transformers import SentenceTransformer\n\n# Select a model to evaluate\nmodel_name = \"sentence-transformers/all-MiniLM-L6-v2\"\n# It's recommended to use mteb.get_model for reproducibility if the model is in MTEB's registry\n# Otherwise, SentenceTransformer can be used directly\nmodel = mteb.get_model(model_name) # Will fall back to SentenceTransformer if not registered in MTEB\n\n# Select tasks to run (e.g., a specific classification task)\ntasks = mteb.get_tasks(tasks=[\"Banking77Classification.v2\"], languages=[\"eng\"])\n\n# Evaluate the model on the selected tasks\nprint(f\"Running evaluation for {model_name} on {len(tasks)} tasks...\")\nresults = mteb.evaluate(model, tasks=tasks)\n\nprint(\"Evaluation complete. Results:\")\nfor task_name, task_results in results.items():\n    print(f\"Task: {task_name}\")\n    print(f\"  Main score: {task_results['main_score']:.4f}\")\n    # Example of accessing detailed metrics\n    if 'accuracy' in task_results['mteb_results']:\n        print(f\"  Accuracy: {task_results['mteb_results']['accuracy']:.4f}\")\n\n# To save results to a specific folder\n# output_folder = f\"./results/{model_name.replace('/', '_')}\"\n# results = mteb.evaluate(model, tasks=tasks, output_folder=output_folder)\n# print(f\"Results saved to: {output_folder}\")","lang":"python","description":"This quickstart demonstrates how to load a pre-trained Sentence Transformer model and evaluate it on a specific MTEB task using the `mteb.evaluate` function. It showcases how to select tasks and retrieve the evaluation results."},"warnings":[{"fix":"Refer to the official MTEB documentation for the updated API, especially focusing on `mteb.evaluate`, `mteb.get_model`, and `mteb.get_tasks`. Ensure your code aligns with the new functional interface rather than directly instantiating the `MTEB` class.","message":"MTEB v2 introduced a large-scale refactor with breaking changes, particularly affecting direct usage of `mteb.MTEB` class and `mteb.load_results` functions. Past minor/patch releases also occasionally introduced breaking changes.","severity":"breaking","affected_versions":"All versions prior to 2.x when upgrading to 2.x; potentially minor versions before 2.x."},{"fix":"Start with smaller task subsets or mini-benchmarks to estimate resource usage. Consider using optimized models or distributed evaluation setups. MTEB also offers caching mechanisms to speed up repeated evaluations.","message":"Evaluating high-performing or large multilingual models on MTEB can be computationally very expensive, requiring significant GPU resources and time, especially for tasks with large document collections like retrieval.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Always perform additional evaluations on your specific domain data to validate model suitability. MTEB can be extended with custom tasks to facilitate this.","message":"Models excelling on the general MTEB leaderboard might underperform on domain-specific data. The benchmark datasets may not perfectly reflect unique domain, user behavior, or query patterns.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Follow the updated submission guidelines on the MTEB GitHub repository or documentation to ensure results are correctly associated with the model implementation.","message":"Directly submitting model results to the MTEB leaderboard by adding metadata to Hugging Face model cards is no longer supported.","severity":"deprecated","affected_versions":"Post-v1.x (Exact version unclear, but mentioned after v2 refactor)"},{"fix":"Replace `model = SentenceTransformer(model_name)` with `model = mteb.get_model(model_name)`. MTEB's function will fall back to `SentenceTransformer` if the model isn't specifically registered.","message":"When evaluating existing models, it is recommended to use `mteb.get_model(\"{model_name}\")` instead of directly using `SentenceTransformer(\"{model_name}\")`. This ensures consistent and reproducible results as it loads the model as MTEB implemented it, accounting for specific normalizations, quantizations, or prompts.","severity":"gotcha","affected_versions":"All versions, particularly relevant for models already on the MTEB leaderboard."}],"env_vars":null,"search_vec":"'1000':37 '2.12.16':52 'across':21 'aim':54 'benchmark':4,10,81 'cadenc':73 'classif':26,90 'cluster':27,91 'compar':64 'comprehens':59 'continu':47 'divers':22 'embed':3,9,17,65,80 'evalu':16,83 'expans':48 'framework':14 'frequent':71 'huggingfac':87 'imag':45 'includ':25 'languag':38 'librari':68 'like':42 'maintain':69 'massiv':1,7 'minor':75 'modal':41 'model':66 'mteb':5,6 'multimod':88 'nlp':23,82 'occur':78 'often':77 'provid':56 'python':13 'releas':72 'reproduc':61 'rerank':29 'retriev':19,28,89 'semant':31,93 'semantic-textual-similar':92 'sentenc':85 'sentence-transform':84 'similar':33,95 'standard':58 'support':35 'system':20 'task':24 'text':2,8,43 'textual':32,94 'transform':86 'updat':76 'various':40 'version':51 'way':62 'week':79","created_at":"2026-04-12T03:41:59.723417+00:00","updated_at":"2026-04-16T16:42:24.837240+00:00","problems":[{"fix":"Ensure `mteb` and its dependencies are up-to-date and correctly installed. It's often resolved by a fresh installation or upgrade: `pip install --upgrade mteb datasets`.","cause":"This error typically occurs when `mteb` or one of its core dependencies, like `datasets`, is not correctly installed, or there's an outdated version mismatch causing certain internal modules or submodules to be missing or reorganized in the installed package.","error":"ModuleNotFoundError: No module named 'mteb.types'"},{"fix":"Check the `mteb` and `sentence-transformers` versions for compatibility. Upgrade both libraries: `pip install --upgrade mteb sentence-transformers`. If the issue persists, the specific model might require custom handling or a different `encode_kwargs` configuration.","cause":"This `TypeError` happens when an embedding model, or its wrapper within `mteb`, is called with a `batch_size` argument, but the model's `encode` method (or the specific implementation being used) does not support or expect this argument. This can be due to an older or incompatible model implementation or an older `mteb` version trying to pass a `batch_size` to a model that doesn't handle it directly.","error":"TypeError: Encoding.encode() got an unexpected keyword argument 'batch_size'"},{"fix":"Reduce the `batch_size` when calling `mteb.evaluate` or directly when encoding. For very large models, consider loading them in lower precision (e.g., `torch_dtype=torch.float16` or 4-bit/8-bit quantization if supported) or using a GPU with more VRAM.","cause":"This common error in deep learning indicates that the GPU ran out of memory while trying to load a model or process a batch of data. This can be caused by using large models, large `batch_size`, or processing long sequences on a GPU with insufficient VRAM.","error":"torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate X MiB (GPU 0; Y GiB total capacity; Z GiB already allocated; A GiB free; B GiB reserved in total by PyTorch)"},{"fix":"Pin the `transformers` library to a version known to be compatible with your `mteb` version, or upgrade `mteb` to its latest version, which might include fixes for newer `transformers` versions: `pip install --upgrade mteb transformers`.","cause":"This `AttributeError` typically arises from incompatibilities between `mteb` and the `transformers` library, especially when `transformers` updates its model output structures (e.g., from raw tensors to `BaseModelOutputWithPooling` objects) or changes the API for specific model types (like audio or cross-encoders). `mteb`'s internal model handling might expect a different attribute or method that no longer exists in the new `transformers` output.","error":"AttributeError: 'BaseModelOutputWithPooling' object has no attribute 'norm'"},{"fix":"Ensure that the dataset configuration (e.g., `corpus`, `queries`, `qrels`) is available for the specific task and language you are running. If you are customizing a task, verify the `AbsTask` implementation or use the `default` configuration if it contains the necessary data. Clearing the Hugging Face dataset cache might also help: `rm -rf ~/.cache/huggingface/datasets`.","cause":"This `ValueError` occurs when `mteb` attempts to load a dataset from the Hugging Face `datasets` library, but the specified dataset configuration (e.g., 'corpus') is not found, and only 'default' is available. This often happens with specific tasks or datasets that have unique configurations or when an old configuration is referenced after a dataset update.","error":"ValueError: BuilderConfig 'corpus' not found. Available: ['default']."}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"2.20.4","cli_name":"mteb","cli_version":"","type":"library","homepage":"https://mteb.ai","github":"https://github.com/embeddings-benchmark/mteb","docs":"https://embeddings-benchmark.github.io/mteb/","changelog":null,"pypi":"https://pypi.org/project/mteb/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["ai-ml","llm-agents"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"failing","verified_at":"2026-06-28","last_verified":"2026-08-29","next_check":"2026-07-28","install_tag":null}}