{"id":13949,"library":"robots-parser","title":"Robots.txt Parser","description":"`robots-parser` is a JavaScript/TypeScript library designed to parse `robots.txt` files according to the draft specification. As of version 3.0.1, it provides robust support for directives such as `User-agent`, `Allow`, `Disallow`, `Sitemap`, `Crawl-delay`, and `Host`, including advanced features like wildcard (`*`) and end-of-line (`$`) matching for paths. The library maintains an active development status, with recent releases addressing critical bug fixes (e.g., HTTPS URL port handling in 3.0.1) and improving compatibility (e.g., using global URL object in 3.0.0, adding TypeScript definitions in 2.4.0). Its primary differentiator is its adherence to the specification and comprehensive feature set for accurately determining URL crawlability for different user agents, making it a reliable choice for web crawlers and SEO tools.","status":"active","version":"3.0.1","language":"javascript","source_language":"en","source_url":"https://github.com/samclarke/robots-parser","tags":["javascript","robots.txt","parser","user-agent","scraper","spider","bot","robots-exclusion-standard","typescript"],"install":[{"cmd":"npm install robots-parser","lang":"bash","label":"npm"},{"cmd":"yarn add robots-parser","lang":"bash","label":"yarn"},{"cmd":"pnpm add robots-parser","lang":"bash","label":"pnpm"}],"dependencies":[],"imports":[{"note":"The primary parsing function is a default export in ESM. For CommonJS, it's the module.exports.","wrong":"import { robotsParser } from 'robots-parser';","symbol":"robotsParser","correct":"import robotsParser from 'robots-parser';"},{"note":"CommonJS `require` syntax for Node.js environments. Node.js v10+ is required since v3.0.0.","wrong":"import robotsParser from 'robots-parser';","symbol":"robotsParser","correct":"const robotsParser = require('robots-parser');"},{"note":"TypeScript type for the instantiated robots parser object. Types were added in v2.4.0.","wrong":"import { RobotsParser } from 'robots-parser';","symbol":"RobotsParser","correct":"import type { RobotsParser } from 'robots-parser';"}],"quickstart":{"code":"import robotsParser from 'robots-parser';\n\nasync function runRobotsParserExample() {\n  const baseUrl = 'http://www.example.com';\n  const robotsTxtContent = `\nUser-agent: *\nDisallow: /dir/\nDisallow: /test.html\nAllow: /dir/test.html\nAllow: /test.html\nCrawl-delay: 1\nSitemap: ${baseUrl}/sitemap.xml\nHost: example.com\n  `.trim();\n\n  // In a real application, you'd typically fetch this content from a URL:\n  // const response = await fetch(`${baseUrl}/robots.txt`);\n  // const fetchedRobotsTxtContent = await response.text();\n\n  const robots = robotsParser(`${baseUrl}/robots.txt`, robotsTxtContent);\n\n  console.log('Is http://www.example.com/test.html allowed for Sams-Bot/1.0?', robots.isAllowed('http://www.example.com/test.html', 'Sams-Bot/1.0'));\n  console.log('Is http://www.example.com/dir/test.html allowed for Sams-Bot/1.0?', robots.isAllowed('http://www.example.com/dir/test.html', 'Sams-Bot/1.0'));\n  console.log('Is http://www.example.com/dir/test2.html disallowed for Sams-Bot/1.0?', robots.isDisallowed('http://www.example.com/dir/test2.html', 'Sams-Bot/1.0'));\n  console.log('Crawl delay for Sams-Bot/1.0:', robots.getCrawlDelay('Sams-Bot/1.0'));\n  console.log('Sitemaps:', robots.getSitemaps());\n  console.log('Preferred Host:', robots.getPreferredHost());\n\n  // Demonstrating undefined for invalid URLs (i.e., not matching the base URL)\n  console.log('Is an out-of-scope URL allowed?', robots.isAllowed('http://www.anotherdomain.com/path', 'Sams-Bot/1.0'));\n}\n\nrunRobotsParserExample();","lang":"typescript","description":"Demonstrates parsing a `robots.txt` string, checking URL crawlability for different user-agents, retrieving sitemaps, crawl delays, and preferred host from the parsed rules. It also illustrates how to handle out-of-scope URLs."},"warnings":[{"fix":"Upgrade Node.js to version 10 or higher, or downgrade `robots-parser` to a 2.x release.","message":"Version 3.0.0 introduced a breaking change by relying on the global `URL` object, removing support for Node.js versions prior to 10. Users on older Node.js environments must upgrade their Node.js version or remain on `robots-parser` v2.x.","severity":"breaking","affected_versions":">=3.0.0"},{"fix":"Handle `undefined` return values explicitly: `if (result === true) { ... } else if (result === false) { ... } else { /* URL out of scope or invalid */ }`","message":"The `isAllowed()` and `isDisallowed()` methods can return `undefined` if the provided URL is not considered valid for the given `robots.txt`'s base URL, rather than a strict boolean. Developers should explicitly check for `undefined` in their logic.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Upgrade to version `3.0.1` or higher to resolve the `https` port default issue.","message":"Prior to version 3.0.1, `https:` URLs without an explicit port would incorrectly default to port `80` instead of `443` during comparison. This could lead to incorrect `isAllowed` or `isDisallowed` results when comparing `https://example.com/` with `https://example.com:443/`.","severity":"breaking","affected_versions":"<3.0.1"},{"fix":"Upgrade to version `2.3.0` or higher to prevent potential errors with specific user-agent strings.","message":"In versions prior to 2.3.0, passing 'constructor' as a user-agent string to `isAllowed()` or `isDisallowed()` could lead to an internal error due to a bug. This edge case was fixed in v2.3.0.","severity":"gotcha","affected_versions":"<2.3.0"},{"fix":"Check for both `undefined` and `-1` when evaluating the result of `getMatchingLineNumber`.","message":"The `getMatchingLineNumber` method can return `-1` if no matching directive is found, or `undefined` if a rule was manually added without an associated line number. It does not consistently return a single type for 'no match' scenarios.","severity":"gotcha","affected_versions":">=1.0.0"}],"env_vars":null,"search_vec":"'2.4.0':91 '3.0.0':86 '3.0.1':23,76 'accord':15 'accur':106 'activ':60 'ad':87 'address':66 'adher':97 'advanc':44 'agent':34,113,130 'allow':35 'bot':133 'bug':68 'choic':118 'compat':79 'comprehens':102 'crawl':39 'crawl-delay':38 'crawlabl':109 'crawler':121 'critic':67 'definit':89 'delay':40 'design':10 'determin':107 'develop':61 'differ':111 'differenti':94 'direct':29 'disallow':36 'draft':18 'e.g':70,80 'end':50 'end-of-lin':49 'exclus':136 'featur':45,103 'file':14 'fix':69 'global':82 'handl':74 'host':42 'https':71 'improv':78 'includ':43 'javascript':125 'javascript/typescript':8 'librari':9,57 'like':46 'line':52 'maintain':58 'make':114 'match':53 'object':84 'pars':12 'parser':2,5,127 'path':55 'port':73 'primari':93 'provid':25 'recent':64 'releas':65 'reliabl':117 'robot':4,135 'robots-exclusion-standard':134 'robots-pars':3 'robots.txt':1,13,126 'robust':26 'scraper':131 'seo':123 'set':104 'sitemap':37 'specif':19,100 'spider':132 'standard':137 'status':62 'support':27 'tool':124 'typescript':88,138 'url':72,83,108 'use':81 'user':33,112,129 'user-ag':32,128 'version':22 'web':120 'wildcard':47","created_at":"2026-04-20T01:57:09.064438+00:00","updated_at":"2026-04-20T01:57:09.064438+00:00","problems":[{"fix":"Upgrade your Node.js environment to version 10.0.0 or higher. Alternatively, downgrade `robots-parser` to a 2.x release if Node.js upgrade is not an option.","cause":"Running `robots-parser` v3.0.0+ on Node.js versions older than 10.0.0, which lack the global `URL` object required by the library.","error":"TypeError: URL is not a constructor"},{"fix":"Ensure `robotsParser` is called with a valid base URL for the `robots.txt` file and correctly formatted `robots.txt` content (as a string). Verify the content for syntax errors or unexpected structures.","cause":"The `robotsParser` function was called with invalid arguments (e.g., incorrect URL format, malformed `robots.txt` content), preventing proper instantiation and returning an invalid object or `undefined`.","error":"TypeError: robots.isAllowed is not a function"},{"fix":"Update `robots-parser` to version `3.0.1` or newer to correctly handle `https` URL port comparisons and ensure accurate permission checks.","cause":"Using `robots-parser` versions prior to 3.0.1, leading to `https:` URLs without explicit ports being incorrectly treated as port `80` instead of `443` during internal comparisons.","error":"Incorrect `isAllowed` or `isDisallowed` result for HTTPS URLs"}],"ecosystem":"npm","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":null,"cli_name":"","cli_version":null,"type":"library","homepage":"https://www.npmjs.com/package/robots-parser","github":"https://github.com/samclarke/robots-parser","docs":null,"changelog":null,"pypi":null,"npm":"https://www.npmjs.com/package/robots-parser","openapi_spec":null,"status_page":null,"smithery":null,"categories":["http-networking"],"base_url":null,"auth_type":null,"provenance":{"verified_status":null,"verified_at":null,"last_verified":"2026-06-17","next_check":"2026-07-18","install_tag":null}}