{"id":3268,"library":"sb-cli","title":"SWE-bench CLI","description":"sb-cli is a command-line interface (CLI) tool designed to interact with the SWE-bench API. It enables users to submit predictions for SWE-bench instances, manage their runs, and track progress. The current version is 0.1.5, and it has an active development cadence driven by the SWE-bench community.","status":"active","version":"0.1.5","language":"python","source_language":"en","source_url":"https://github.com/swe-bench/sb-cli","tags":["cli","swe-bench","machine-learning","evaluation","tool"],"install":[{"cmd":"pip install sb-cli","lang":"bash","label":"Install sb-cli"}],"dependencies":[],"imports":[{"wrong":"from sb_cli import app","symbol":"app","correct":"from sb_cli import app"}],"quickstart":{"code":"import subprocess\nimport os\n\n# NOTE: This quickstart demonstrates submitting a run via the CLI.\n# You must first configure your SWE-bench API token using `sb config --api-token YOUR_TOKEN`\n# and start a run using `sb start` before submitting.\n\n# Placeholder for your instance and prediction paths\ninstance_path = os.environ.get('SB_INSTANCE_PATH', 'path/to/your/swe_bench_instance.json')\nprediction_path = os.environ.get('SB_PREDICTION_PATH', 'path/to/your/prediction.json')\n\nif not os.path.exists(instance_path) or not os.path.exists(prediction_path):\n    print(f\"Warning: Placeholder paths used. Please replace '{instance_path}' and '{prediction_path}' \"\n          \"with actual file paths for a successful submission.\")\n    print(\"You may need to run 'sb config' and 'sb start' first.\")\n    # Exit or mock for demonstration if files don't exist\n    # For a real run, ensure these paths are valid.\n    # For this example, we will just print the command without executing if paths are missing.\n    command = [\n        \"sb\",\n        \"submit\",\n        \"--instance-path\", instance_path,\n        \"--prediction-path\", prediction_path,\n        \"--log-dir\", \"./sb_logs\" # Optional: specify a log directory\n    ]\n    print(f\"Mock command to execute: {' '.join(command)}\")\nelse:\n    print(\"Attempting to submit SWE-bench prediction...\")\n    command = [\n        \"sb\",\n        \"submit\",\n        \"--instance-path\", instance_path,\n        \"--prediction-path\", prediction_path,\n        \"--log-dir\", \"./sb_logs\" # Optional: specify a log directory\n    ]\n    try:\n        result = subprocess.run(command, capture_output=True, text=True, check=True)\n        print(\"Submission successful!\")\n        print(\"STDOUT:\", result.stdout)\n        if result.stderr: print(\"STDERR:\", result.stderr)\n    except subprocess.CalledProcessError as e:\n        print(\"Submission failed!\")\n        print(\"STDOUT:\", e.stdout)\n        print(\"STDERR:\", e.stderr)\n    except FileNotFoundError:\n        print(\"Error: 'sb' command not found. Is sb-cli installed and in your PATH?\")","lang":"python","description":"This quickstart demonstrates how to programmatically execute the `sb submit` command using Python's `subprocess` module. Before running, ensure `sb-cli` is installed and configured with your SWE-bench API token (`sb config --api-token YOUR_TOKEN`). You also need to have started a run using `sb start` and provide valid paths to your SWE-bench instance and prediction files."},"warnings":[{"fix":"Upgrade your Python environment to 3.10 or later (`python -m pip install --upgrade python` or use `pyenv`, `conda`).","message":"sb-cli requires Python 3.10 or newer. Attempting to use it with older Python versions will result in installation or runtime errors.","severity":"gotcha","affected_versions":"<0.1.0 (all versions)"},{"fix":"Run `sb config --api-token YOUR_API_TOKEN` once to set up your authentication. The token is typically obtained from the SWE-bench website.","message":"Submitting predictions requires an API token, which must be configured using `sb config --api-token YOUR_TOKEN`. Failure to do so will result in authentication errors when interacting with the SWE-bench API.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure that the `instance_path` and `prediction_path` arguments point to valid, existing, and readable JSON files on your filesystem.","message":"The `sb submit` command relies on local file paths for both the SWE-bench instance and prediction files. Incorrect or inaccessible paths will lead to submission failures.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Interact with sb-cli through `subprocess` calls in Python scripts, mirroring how you would use it directly from the command line. Avoid relying on direct imports of internal modules unless specifically for testing or advanced scenarios.","message":"sb-cli is designed primarily as a command-line interface. While parts of its internal Python code can be imported, there isn't a high-level, stable, or officially documented programmatic Python API for general library use. Most interactions are expected to occur via shell commands.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"search_vec":"'0.1.5':46 'activ':51 'api':24 'bench':3,23,34,59,64 'cadenc':53 'cli':4,7,14,61 'command':11 'command-lin':10 'communiti':60 'current':43 'design':16 'develop':52 'driven':54 'enabl':26 'evalu':68 'instanc':35 'interact':18 'interfac':13 'learn':67 'line':12 'machin':66 'machine-learn':65 'manag':36 'predict':30 'progress':41 'run':38 'sb':6 'sb-cli':5 'submit':29 'swe':2,22,33,58,63 'swe-bench':1,21,32,57,62 'tool':15,69 'track':40 'user':27 'version':44","created_at":"2026-04-11T09:28:14.079220+00:00","updated_at":"2026-04-16T21:21:56.959417+00:00","problems":{"verify_error":"Traceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"/tmp/tmpeg2d8tqy/venv/lib/python3.12/site-packages/sb_cli/__init__.py\", line 5, in <module>\n    from . import (\n  File \"/tmp/tmpeg2d8tqy/venv/lib/python3.12/site-packages/sb_cli/submit.py\", line 7, in <module>\n    from"},"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.1.5","cli_name":"sb","cli_version":"sh: 1: sb: not found","type":"library","homepage":null,"github":"https://github.com/swe-bench/sb-cli","docs":null,"changelog":null,"pypi":"https://pypi.org/project/sb-cli/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["devops","http-networking"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-07-03","last_verified":"2026-08-29","next_check":"2026-07-10","install_tag":null}}