{"id":7745,"library":"spider-client","title":"Spider Cloud Python SDK","description":"The `spider-client` is a Python SDK for integrating with the Spider Cloud API, providing tools for web scraping, large-scale crawling, link extraction, and taking screenshots. It is designed to efficiently collect data, often formatted for compatibility with Language Models (LLMs), leveraging a Rust-based engine optimized for AI that supports concurrent operations, streaming, and headless Chrome rendering. The library is actively maintained, with frequent updates, and the current version is 0.1.88.","status":"active","version":"0.1.88","language":"python","source_language":"en","source_url":"https://github.com/spider-rs/spider-clients/tree/main/python","tags":["web scraping","crawling","api client","ai","llm","data extraction","streaming"],"install":[{"cmd":"pip install spider-client","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"Used for iterative parsing of large JSON streams, added in v0.1.37 to improve handling of substantial data payloads.","package":"ijson","optional":false}],"imports":[{"wrong":"from spider_client import Spider","symbol":"Spider","correct":"from spider import Spider"},{"symbol":"AsyncSpider","correct":"from spider import AsyncSpider"},{"symbol":"spider","correct":"from spider import spider"}],"quickstart":{"code":"import os\nfrom spider_client import Spider\n\n# Retrieve API key from environment variable or replace with your actual key\n# Get an API key from https://spider.cloud\napi_key = os.environ.get('SPIDER_API_KEY', 'YOUR_SPIDER_API_KEY')\n\nif not api_key or api_key == 'YOUR_SPIDER_API_KEY':\n    print(\"WARNING: SPIDER_API_KEY not set. Please set it as an environment variable or pass to Spider(api_key=...).\\nSkipping API call.\")\nelse:\n    app = Spider(api_key=api_key)\n\n    url_to_scrape = 'https://example.com'\n    try:\n        scraped_data = app.scrape_url(url_to_scrape)\n        print(f\"Successfully scraped data from {url_to_scrape}:\")\n        print(scraped_data)\n    except Exception as e:\n        print(f\"An error occurred during scraping: {e}\")","lang":"python","description":"This quickstart initializes the Spider client and performs a basic URL scrape. It demonstrates how to configure the API key, either via an environment variable or direct instantiation, and handles a simple scraping operation. Obtain your API key from spider.cloud."},"warnings":[{"fix":"Ensure your Python environment is up-to-date (preferably Python 3.9+ for native pipe operator syntax) and review any custom type hint definitions that might conflict with the change. If issues persist, check the official GitHub for detailed migration guides.","message":"The `v0.1.37` release included a fix that 'removed pipe operator' (`fix(python): removed pipe operator`). This likely pertains to changes in type hinting syntax or internal handling of union types, which might affect compatibility with specific older Python versions or code relying on a previous internal implementation.","severity":"breaking","affected_versions":"<0.1.37"},{"fix":"Obtain an API key from spider.cloud and set it as an environment variable `SPIDER_API_KEY` or pass it directly to the `Spider` constructor: `app = Spider(api_key='YOUR_API_KEY')`.","message":"The library requires an API key for authentication with the Spider Cloud API. Requests without a valid API key will fail with authentication errors.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Utilize the `stream=True` parameter in methods like `crawl_url` and `scrape_url`, and process the response iteratively. The library internally uses `ijson` for efficient streaming. Follow examples for processing chunks or streaming directly.","message":"When dealing with large JSON responses or streaming data, direct `json.loads()` on the entire response might lead to `json.JSONDecodeError` due to incomplete data or excessive memory usage.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"search_vec":"'0.1.88':80 'activ':70 'ai':57,86 'api':19,84 'base':53 'chrome':65 'client':8,85 'cloud':2,18 'collect':39 'compat':44 'concurr':60 'crawl':28,83 'current':77 'data':40,88 'design':36 'effici':38 'engin':54 'extract':30,89 'format':42 'frequent':73 'headless':64 'integr':14 'languag':46 'larg':26 'large-scal':25 'leverag':49 'librari':68 'link':29 'llm':87 'llms':48 'maintain':71 'model':47 'often':41 'oper':61 'optim':55 'provid':20 'python':3,11 'render':66 'rust':52 'rust-bas':51 'scale':27 'scrape':24,82 'screenshot':33 'sdk':4,12 'spider':1,7,17 'spider-cli':6 'stream':62,90 'support':59 'take':32 'tool':21 'updat':74 'version':78 'web':23,81","created_at":"2026-04-16T14:11:34.136789+00:00","updated_at":"2026-04-16T14:11:34.136789+00:00","problems":null,"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.1.88","cli_name":"spider-client","cli_version":"sh: 1: spider-client: not found","type":"library","homepage":"https://spider.cloud","github":"https://github.com/spider-rs/spider-clients","docs":null,"changelog":null,"pypi":"https://pypi.org/project/spider-client/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["http-networking","ai-ml","llm-agents","data"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-07-03","last_verified":"2026-07-03","next_check":"2026-08-02","install_tag":null}}