{"id":1351,"library":"azure-kusto-ingest","title":"Azure Kusto Ingest Client","description":"The `azure-kusto-ingest` library provides a client for ingesting data into Azure Data Explorer (Kusto) clusters. It supports queued ingestion (batching for high throughput) and streaming ingestion (low latency). The current version is 6.0.3, with frequent bug fix releases and occasional major versions aligning with Python and Azure SDK ecosystem changes. Version 6.0.3 introduced allowing transformation functions for CSV and SCSV formats.","status":"active","version":"6.0.3","language":"python","source_language":"en","source_url":"https://github.com/Azure/azure-kusto-python","tags":["azure","kusto","azure data explorer","data ingestion","cloud","analytics"],"install":[{"cmd":"pip install azure-kusto-ingest","lang":"bash","label":"Base installation"},{"cmd":"pip install azure-kusto-ingest[pandas]","lang":"bash","label":"With Pandas support (for DataFrame ingestion)"}],"dependencies":[{"reason":"Required for `ingest_from_dataframe` functionality, installed via `[pandas]` extra.","package":"pandas","optional":true}],"imports":[{"note":"Used to construct connection strings, part of the core Kusto data library.","symbol":"KustoConnectionStringBuilder","correct":"from azure.kusto.data import KustoConnectionStringBuilder"},{"note":"Main client for queued (batch) data ingestion.","symbol":"KustoIngestClient","correct":"from azure.kusto.ingest import KustoIngestClient"},{"note":"Client for low-latency streaming data ingestion.","symbol":"KustoStreamingIngestClient","correct":"from azure.kusto.ingest import KustoStreamingIngestClient"},{"note":"Configures ingestion behavior like target database/table, data format, and mapping.","symbol":"IngestionProperties","correct":"from azure.kusto.ingest import IngestionProperties"},{"note":"Enum for specifying the format of the ingested data (e.g., CSV, TSV, JSON).","symbol":"DataFormat","correct":"from azure.kusto.data import DataFormat"},{"note":"Used for ingesting data directly from Azure Blob Storage.","symbol":"BlobDescriptor","correct":"from azure.kusto.ingest import BlobDescriptor"}],"quickstart":{"code":"import os\nimport io\nfrom azure.kusto.data import KustoConnectionStringBuilder, DataFormat\nfrom azure.kusto.ingest import KustoIngestClient, IngestionProperties\n\n# Configuration from environment variables (replace with your actual values)\nKUSTO_CLUSTER_URL = os.environ.get('KUSTO_CLUSTER_URL', 'https://yourcluster.region.kusto.windows.net')\nKUSTO_DATABASE = os.environ.get('KUSTO_DATABASE', 'yourdatabase')\nKUSTO_TABLE = os.environ.get('KUSTO_TABLE', 'yourtable')\n\n# Ensure environment variables are set or replace with actual connection string details\n# For AAD app authentication:\n# KCSB = KustoConnectionStringBuilder.with_aad_application_key(\n#     KUSTO_CLUSTER_URL, os.environ.get('KUSTO_CLIENT_ID', ''), os.environ.get('KUSTO_CLIENT_SECRET', ''), os.environ.get('KUSTO_TENANT_ID', '')\n# )\n# For Azure CLI authentication:\nKCSB = KustoConnectionStringBuilder.with_az_cli_authentication(KUSTO_CLUSTER_URL)\n\n# Create an Ingest Client\ningest_client = KustoIngestClient(KCSB)\n\n# Define Ingestion Properties\ningestion_properties = IngestionProperties(\n    database=KUSTO_DATABASE,\n    table=KUSTO_TABLE,\n    data_format=DataFormat.CSV,\n    # For real-time monitoring of ingestion status, set flush_immediately=True\n    # but be aware of performance implications for large batches.\n    flush_immediately=False\n)\n\n# Sample data as a CSV string\ndata_rows = [\n    \"id,name,value\",\n    \"1,TestItemA,100\",\n    \"2,TestItemB,200\"\n]\ndata_csv = \"\\n\".join(data_rows)\n\n# Ingest data from an in-memory stream\nprint(f\"Attempting to ingest data into {KUSTO_DATABASE}.{KUSTO_TABLE}...\")\nwith io.StringIO(data_csv) as stream:\n    result = ingest_client.ingest_from_stream(stream, ingestion_properties)\n\n# For queued ingestion, the result indicates submission, not completion.\n# Actual status must be monitored in Azure Data Explorer using .show commands.\nprint(\"Ingestion job submitted. Check Azure Data Explorer for status details.\")\n# print(f\"Ingestion result object: {result}\") # Uncomment to see the result object\n","lang":"python","description":"This quickstart demonstrates how to perform queued ingestion of in-memory CSV data into an Azure Data Explorer table. It uses Azure CLI authentication for simplicity but can be adapted for other authentication methods. The example sets up the Kusto client, defines ingestion properties, and sends a small CSV string as a stream. Remember to set `KUSTO_CLUSTER_URL`, `KUSTO_DATABASE`, and `KUSTO_TABLE` environment variables."},"warnings":[{"fix":"Upgrade your Python environment to 3.9 or higher. If you must use an older Python version, pin `azure-kusto-ingest` to `<6.0.0` (e.g., `azure-kusto-ingest<6.0.0`).","message":"Starting with version 6.0.0, the minimum supported Python version for `azure-kusto-ingest` (and other Azure SDKs) is Python 3.9. Prior versions supported older Python versions (e.g., v5.x supported Python 3.8, v4.x supported Python 3.7).","severity":"breaking","affected_versions":">=6.0.0"},{"fix":"Review your `KustoConnectionStringBuilder` usage. Use the builder methods (e.g., `with_aad_managed_identity`) instead of relying on parsing specific keyword arguments in the connection string directly. If using older keyword-based strings, downgrade to `<5.0.0`.","message":"Version 5.0.0 introduced breaking changes to `KustoConnectionStringBuilder` keywords, aligning them with other Azure SDKs. Keywords like `msi_auth`, `msi_authentication`, `msi_params`, and `msi_type` were removed from direct parsing.","severity":"breaking","affected_versions":">=5.0.0"},{"fix":"Always use the latest version of `azure-kusto-ingest` with the latest compatible Pandas version. If encountering issues, specifically check the changelog for fixes related to `dataframe_from_result_table` and `ingest_from_dataframe` in your library version relative to your Pandas version. Pinning `pandas` to a known working version might be necessary if unable to update `azure-kusto-ingest`.","message":"Ingesting data from Pandas DataFrames (using `ingest_from_dataframe`) has seen multiple bug fixes across versions related to datetime columns, null values, and specific Pandas versions (e.g., Pandas 3.0). Ensure compatibility.","severity":"gotcha","affected_versions":"<6.0.2"},{"fix":"Upgrade to version `6.0.1` or higher for improved throttling event handling in Managed Streaming. If using streaming ingestion, ensure your cluster is adequately scaled for the expected ingestion rate.","message":"Managed Streaming Ingestion can encounter throttling events. While version 6.0.1 included a fix to handle these events more gracefully, users on older versions might experience issues with managed streaming stability under high load.","severity":"gotcha","affected_versions":"<6.0.1"},{"fix":"Update your code to import `KustoClient` instead of `KustoIngestClient`. For example, `from azure.kusto.ingest import KustoClient`. If you need to support older versions, consider conditional imports or pin the library to `<6.0.0`.","message":"The class `KustoIngestClient` was renamed to `KustoClient` starting from version 6.0.0. Attempting to import `KustoIngestClient` from `azure.kusto.ingest` in versions `>=6.0.0` will result in an `ImportError`.","severity":"breaking","affected_versions":">=6.0.0"},{"fix":"Update your code to import and use either `QueuedKustoIngestClient` or `StreamKustoIngestClient` from `azure.kusto.ingest` instead of `KustoIngestClient`. For example, change `from azure.kusto.ingest import KustoIngestClient` to `from azure.kusto.ingest import QueuedKustoIngestClient` or `StreamKustoIngestClient` and adjust the client instantiation accordingly.","message":"Starting with version 6.0.0, the `KustoIngestClient` class was removed from the `azure.kusto.ingest` top-level module. Users should transition to `QueuedKustoIngestClient` for queued ingestion or `StreamKustoIngestClient` for streaming ingestion, depending on their use case.","severity":"breaking","affected_versions":">=6.0.0"}],"env_vars":null,"search_vec":"'6.0.3':40,59 'align':50 'allow':61 'analyt':77 'azur':1,7,18,54,69,71 'azure-kusto-ingest':6 'batch':27 'bug':43 'chang':57 'client':4,13 'cloud':76 'cluster':22 'csv':65 'current':37 'data':16,19,72,74 'ecosystem':56 'explor':20,73 'fix':44 'format':68 'frequent':42 'function':63 'high':29 'ingest':3,9,15,26,33,75 'introduc':60 'kusto':2,8,21,70 'latenc':35 'librari':10 'low':34 'major':48 'occasion':47 'provid':11 'python':52 'queu':25 'releas':45 'scsv':67 'sdk':55 'stream':32 'support':24 'throughput':30 'transform':62 'version':38,49,58","created_at":"2026-04-09T03:44:12.655123+00:00","updated_at":"2026-04-15T22:22:43.835640+00:00","problems":[{"fix":"Ensure the `azure-kusto-ingest` package is installed in the correct Python environment. If using a virtual environment, activate it before installing. For platforms like Databricks, ensure the library is installed for the cluster's Python environment. \n`pip install azure-kusto-ingest`","cause":"This error occurs when the `azure-kusto-ingest` package (or its dependencies within the `azure` namespace) is not properly installed or accessible in the Python environment where the code is being run. This is particularly common in isolated environments like Databricks notebooks or virtual environments where the installation might be in a different interpreter or not linked correctly.","error":"ModuleNotFoundError: No module named 'azure'"},{"fix":"Check the official `azure-kusto-ingest` documentation for the correct import path of `DataFormat` in your installed version. It is often located in a submodule like `azure.kusto.data.data_format`. \n`from azure.kusto.data.data_format import DataFormat`","cause":"This error typically indicates that the `DataFormat` enum (or a similar constant/class) has been moved, renamed, or removed from the direct `azure.kusto.ingest` module in a newer version of the library, or that a user is trying to import it incorrectly.","error":"ImportError: cannot import name 'DataFormat' from 'azure.kusto.ingest'"},{"fix":"Verify that your Azure AD application or managed identity has the necessary 'Ingestor' permissions on the target database in Azure Data Explorer. Double-check the client ID, client secret (or certificate), tenant ID, and the Kusto cluster URI in your connection string.","cause":"This error occurs when the client fails to authenticate with the Azure Data Explorer cluster. Common reasons include incorrect credentials (e.g., Azure AD application ID, secret, or certificate), insufficient permissions for the authenticating identity (e.g., 'ingestor' role not assigned), or an incorrect cluster endpoint.","error":"KustoAuthenticationError"},{"fix":"Ensure the table name provided to the ingestion client is correct and that the table has been created in your Azure Data Explorer database. You can create a table using Kusto Query Language (KQL) commands in the Azure Data Explorer web UI or programmatically.","cause":"This error indicates that the target table specified in the ingestion command does not exist in the Azure Data Explorer database, or its name is misspelled.","error":"Entity 'table name that doesn't exist' of kind 'Table' wasn't found."},{"fix":"Review your data source and ensure its schema (column names, types, order) aligns with the target Kusto table's schema. If using ingestion mappings, verify they are correctly defined and reference the appropriate columns. For CSV files, ensure all rows have a consistent number of fields. You may need to modify your ingestion mapping or the table schema to align with the incoming data.","cause":"These errors indicate a mismatch between the schema of the data being ingested and the schema of the target table in Azure Data Explorer, or an inconsistency within the input data itself (e.g., a CSV file with varying numbers of columns per row). This can also occur if ingestion mappings are incorrect or outdated.","error":"Query schema doesn't match table schema / Inconsistent number of fields in the input records."}],"ecosystem":"pypi","meta_description":null,"install_score":100,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"6.0.4","cli_name":"","cli_version":null,"type":"library","homepage":"https://docs.microsoft.com/en-us/azure/data-explorer","github":"https://github.com/Azure/azure-kusto-python","docs":null,"changelog":null,"pypi":"https://pypi.org/project/azure-kusto-ingest/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["azure","database"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-06-28","next_check":"2026-07-28","install_tag":"verified"}}