{"id":1839,"library":"hdfs","title":"HdfsCLI: API and Command Line Interface for HDFS","description":"HdfsCLI provides a Python API and command-line interface for interacting with Hadoop HDFS via the WebHDFS (and HttpFS) API. It supports both secure and insecure clusters, offering Python 3 bindings for common HDFS operations. The library includes optional extensions for handling Avro files, Pandas DataFrames, and Kerberos authentication. The current version, 2.7.3, was released on October 12, 2023, indicating active maintenance.","status":"active","version":"2.7.3","language":"python","source_language":"en","source_url":"https://github.com/mtth/hdfs","tags":["hdfs","hadoop","filesystem","client","webhdfs","httpfs","bigdata"],"install":[{"cmd":"pip install hdfs","lang":"bash","label":"Core library"},{"cmd":"pip install hdfs[avro,dataframe,kerberos]","lang":"bash","label":"With optional extensions"}],"dependencies":[{"reason":"Required for the 'avro' extension to read and write Avro files.","package":"fastavro","optional":true},{"reason":"Required for the 'dataframe' extension to load and save Pandas DataFrames.","package":"pandas","optional":true},{"reason":"Required for the 'kerberos' extension to enable Kerberos authenticated clusters.","package":"requests-kerberos","optional":true}],"imports":[{"note":"The default and simplest client for insecure HDFS clusters.","symbol":"InsecureClient","correct":"from hdfs.client import InsecureClient"},{"note":"The base client class, InsecureClient is a subclass. Often used via `Client.from_alias()`.","symbol":"Client","correct":"from hdfs.client import Client"},{"note":"Used for token-based authentication with HDFS.","symbol":"TokenClient","correct":"from hdfs.client import TokenClient"},{"note":"Used for Kerberos authenticated clusters, requires the 'kerberos' extension.","symbol":"KerberosClient","correct":"from hdfs.ext.kerberos import KerberosClient"}],"quickstart":{"code":"import os\nfrom hdfs.client import InsecureClient\n\nHDFS_NAMENODE_URL = os.environ.get('HDFS_NAMENODE_URL', 'http://localhost:50070')\nHDFS_USER = os.environ.get('HDFS_USER', 'guest') # Or a specific HDFS user\n\ntry:\n    client = InsecureClient(HDFS_NAMENODE_URL, user=HDFS_USER)\n    print(f\"Connected to HDFS at {HDFS_NAMENODE_URL} as user {HDFS_USER}\")\n\n    # Example: Create a file\n    hdfs_path = '/user/temp/my_test_file.txt'\n    local_data = b'Hello, HdfsCLI world!'\n    with client.write(hdfs_path, encoding='utf-8', overwrite=True) as writer:\n        writer.write(local_data.decode('utf-8'))\n    print(f\"Successfully wrote to {hdfs_path}\")\n\n    # Example: List contents of a directory\n    parent_dir = os.path.dirname(hdfs_path)\n    if parent_dir == '': parent_dir = '/' # handle root edge case\n    print(f\"Contents of {parent_dir}:\")\n    for item in client.list(parent_dir):\n        print(f\"- {item}\")\n    \n    # Example: Read the file back\n    with client.read(hdfs_path, encoding='utf-8') as reader:\n        read_data = reader.read()\n    print(f\"Read from {hdfs_path}: {read_data}\")\n\n    # Example: Delete the file\n    client.delete(hdfs_path)\n    print(f\"Successfully deleted {hdfs_path}\")\n\nexcept Exception as e:\n    print(f\"An error occurred: {e}\")\n    print(\"Please ensure HDFS is running and HDFS_NAMENODE_URL/HDFS_USER are correctly configured.\")","lang":"python","description":"This quickstart demonstrates how to establish a connection to an HDFS Namenode using `InsecureClient`, write a simple file, list directory contents, read the file back, and then delete it. It uses environment variables for the Namenode URL and user for flexibility. Ensure your HDFS cluster is running and accessible at the specified URL."},"warnings":[{"fix":"Upgrade your Python environment to 3.7 or newer. If you must use Python 2, you'll need to use an older version of the `hdfs` library (e.g., `hdfs<2.0.0`), but this is not recommended due to lack of maintenance.","message":"HdfsCLI version 2.x and above has dropped official support for Python 2.x. It is compatible with Python 3.7+.","severity":"breaking","affected_versions":"<2.0.0"},{"fix":"When calling `client.write()`, include `overwrite=True` in the arguments if you intend to replace an existing file (e.g., `client.write(path, overwrite=True)`).","message":"By default, `client.write()` will raise an `HdfsError` if trying to write to an existing path. To overwrite an existing file, you must explicitly set `overwrite=True`.","severity":"gotcha","affected_versions":"All"},{"fix":"To delete a directory and its contents, use `client.delete(path, recursive=True)`. Consider `skip_trash=False` (requires Hadoop 2.9+) if you want files to go to trash instead of being permanently deleted.","message":"Deleting a non-empty directory without `recursive=True` will raise an `HdfsError`. This is a safety mechanism.","severity":"gotcha","affected_versions":"All"},{"fix":"Ensure you have a `~/.hdfscli.cfg` file (or `HDFSCLI_CONFIG` environment variable pointing to one) with valid alias definitions, including `url` and optional `user` or `client` (e.g., `KerberosClient`).","message":"Using `Client.from_alias()` relies on a configuration file (default: `~/.hdfscli.cfg`) which defines cluster connection details. Without proper configuration, this method will fail.","severity":"gotcha","affected_versions":"All"},{"fix":"Install the kerberos extension (`pip install hdfs[kerberos]`). Ensure your `krb5.conf` is correctly configured and you have a valid Kerberos ticket. Refer to the HdfsCLI documentation for detailed Kerberos setup instructions.","message":"The `KerberosClient` requires the `hdfs[kerberos]` extra to be installed and proper Kerberos configuration on the client machine and HDFS cluster. Misconfiguration often leads to authentication errors.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"search_vec":"'12':67 '2.7.3':62 '2023':68 '3':39 'activ':70 'api':2,13,29 'authent':58 'avro':52 'bigdata':78 'bind':40 'client':75 'cluster':36 'command':4,16 'command-lin':15 'common':42 'current':60 'datafram':55 'extens':49 'file':53 'filesystem':74 'hadoop':22,73 'handl':51 'hdfs':8,23,43,72 'hdfscli':1,9 'httpfs':28,77 'includ':47 'indic':69 'insecur':35 'interact':20 'interfac':6,18 'kerbero':57 'librari':46 'line':5,17 'mainten':71 'octob':66 'offer':37 'oper':44 'option':48 'panda':54 'provid':10 'python':12,38 'releas':64 'secur':33 'support':31 'version':61 'via':24 'webhdf':26,76","created_at":"2026-04-09T05:07:30.160337+00:00","updated_at":"2026-04-16T15:33:21.855265+00:00","problems":[{"fix":"Verify the HDFS Namenode host and port (e.g., from `core-site.xml`'s `fs.default.name` property) and ensure they are correctly specified when initializing the `hdfs.InsecureClient` or `hdfs.Client`. Ensure the HDFS service is running and accessible from the client machine. \n```python\nfrom hdfs import InsecureClient\n\n# Replace 'your_hdfs_namenode_host' and 'your_hdfs_port' with actual values\n# Example: 'http://localhost:9870' or 'http://your_namenode:50070'\nclient = InsecureClient('http://your_hdfs_namenode_host:your_hdfs_port', user='your_hdfs_user')\n# Or, for a secure cluster with Kerberos (requires hdfs[kerberos] installed and kinit):\n# from hdfs.ext.kerberos import KerberosClient\n# client = KerberosClient('http://your_hdfs_namenode_host:your_hdfs_port', user='your_hdfs_user')\n```","cause":"This error typically occurs when the HDFS client cannot establish a connection to the Hadoop Distributed File System, often due to incorrect host, port, or user parameters in the client initialization.","error":"OSError: HDFS connection failed"},{"fix":"Check that the HDFS Namenode and DataNode services are running, the host and port are correct and reachable from your client machine, and no firewall is blocking the WebHDFS port (typically 50070 or 9870 for the Namenode UI/WebHDFS endpoint, or 50075 for DataNode). Use `curl` to test connectivity to the WebHDFS endpoint from your client machine. \n```bash\n# Example: Test WebHDFS API endpoint\ncurl -i 'http://your_hdfs_namenode_host:your_hdfs_port/webhdfs/v1/?op=GETHOMEDIRECTORY'\n```\nIf `curl` also fails, the issue is with network connectivity or the HDFS cluster setup. If `curl` succeeds, verify the `hdfs` client initialization parameters.","cause":"This error indicates that the Python `hdfs` client, which uses WebHDFS, failed to establish an HTTP connection to the specified HDFS endpoint after multiple retries. This can be caused by an incorrect host/port, network issues (e.g., firewall blocking the port), or the WebHDFS service not running on the Hadoop cluster.","error":"ConnectionError: HTTPConnectionPool(host='...', port=...): Max retries exceeded with url:"},{"fix":"Ensure you are using an up-to-date version of the `hdfs` library (version 2.7.3 or newer is recommended) and use the correct import pattern for `InsecureClient` or `Client`. If upgrading, remove the old version first. \n```bash\npip uninstall hdfs\npip install hdfs\n```\nThen, use the standard import: \n```python\nfrom hdfs import InsecureClient\n# client = InsecureClient('http://namenode_host:port', user='your_user')\n# For Kerberos:\n# from hdfs.ext.kerberos import KerberosClient\n# client = KerberosClient('http://namenode_host:port', user='your_user')\n```","cause":"These errors usually stem from using an outdated version of the `hdfs` library or incorrect import statements based on older API versions. The `hdfs` library's structure or common usage patterns might have changed.","error":"AttributeError: module 'hdfs' has no attribute 'client' OR ImportError: cannot import name 'config' from 'hdfs'"},{"fix":"For Kerberized clusters, ensure you have obtained a valid Kerberos ticket before running your Python application. This typically involves using the `kinit` command. Additionally, ensure the `hdfs[kerberos]` extra is installed if using Kerberos. \n```bash\nkinit your_user@YOUR.REALM\n# Then, in your Python code:\npip install 'hdfs[kerberos]'\nfrom hdfs.ext.kerberos import KerberosClient\nclient = KerberosClient('http://your_hdfs_namenode_host:your_hdfs_port')\n# The 'user' parameter is often not needed with KerberosClient as it's derived from the kinit ticket.\n```","cause":"This `HdfsError` occurs when trying to connect to a secure (Kerberized) HDFS cluster without proper authentication credentials, such as a valid Kerberos ticket.","error":"Authentication failure. Check your credentials."}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"2.7.3","cli_name":"hdfs","cli_version":"sh: 1: hdfs: not found","type":"library","homepage":"https://hdfscli.readthedocs.io","github":null,"docs":null,"changelog":null,"pypi":"https://pypi.org/project/hdfs/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["database","data","serialization"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-06-28","next_check":"2026-07-28","install_tag":null}}