{"id":5499,"library":"splink","title":"Splink","description":"Splink is a Python package for fast, accurate, and scalable probabilistic record linkage (entity resolution). It enables users to deduplicate and link records from datasets that lack unique identifiers, leveraging unsupervised learning based on the Fellegi-Sunter model. Splink supports various SQL backends like DuckDB, Apache Spark, and AWS Athena, allowing it to scale to datasets of 100 million records or more, and provides a suite of interactive visualizations for model understanding and diagnostics.","status":"active","version":"4.0.16","language":"python","source_language":"en","source_url":"https://github.com/moj-analytical-services/splink","tags":["data linkage","entity resolution","deduplication","probabilistic matching","big data","sql","data science"],"install":[{"cmd":"pip install splink","lang":"bash","label":"Base Install (includes DuckDB and SQLite)"},{"cmd":"pip install 'splink[spark]'","lang":"bash","label":"For Apache Spark backend"},{"cmd":"pip install 'splink[athena]'","lang":"bash","label":"For AWS Athena backend"},{"cmd":"pip install 'splink[postgres]'","lang":"bash","label":"For PostgreSQL backend"}],"dependencies":[{"reason":"Default high-performance SQL backend, bundled with base install.","package":"duckdb"},{"reason":"Bundled SQL backend, suited for smaller datasets.","package":"sqlite3"},{"reason":"Optional backend for big data processing (installed with 'splink[spark]').","package":"pyspark"},{"reason":"Optional backend for AWS Athena (installed with 'splink[athena]'). Note: support is being dropped in v5.","package":"pyathena"},{"reason":"Optional backend for PostgreSQL (installed with 'splink[postgres]').","package":"psycopg2-binary"},{"reason":"Used for SQL transpilation to ensure compatibility across multiple SQL engines.","package":"sqlglot"}],"imports":[{"symbol":"Linker","correct":"from splink import Linker"},{"symbol":"SettingsCreator","correct":"from splink import SettingsCreator"},{"symbol":"block_on","correct":"from splink import block_on"},{"symbol":"DuckDBAPI","correct":"from splink import DuckDBAPI"},{"symbol":"splink_datasets","correct":"from splink import splink_datasets"},{"symbol":"cl","correct":"import splink.comparison_library as cl"}],"quickstart":{"code":"import splink.comparison_library as cl\nfrom splink import DuckDBAPI, Linker, SettingsCreator, block_on, splink_datasets\n\ndb_api = DuckDBAPI()\ndf = splink_datasets.fake_1000\n\nsettings = SettingsCreator(\n    link_type=\"dedupe_only\",\n    comparisons=[\n        cl.NameComparison(\"first_name\"),\n        cl.JaroAtThresholds(\"surname\"),\n        cl.DateOfBirthComparison(\"dob\", input_is_string=True),\n        cl.ExactMatch(\"city\").configure(term_frequency_adjustments=True),\n        cl.EmailComparison(\"email\"),\n    ],\n    blocking_rules_to_generate_predictions=[\n        block_on(\"first_name\", \"dob\"),\n        block_on(\"surname\"),\n    ]\n)\n\nlinker = Linker(df, settings, db_api)\n\nlinker.training.estimate_probability_two_random_records_match(\n    [block_on(\"first_name\", \"surname\")], recall=0.7\n)\nlinker.training.estimate_u_using_random_sampling(max_pairs=1e6)\nlinker.training.estimate_parameters_using_expectation_maximisation(\n    block_on(\"first_name\", \"surname\")\n)\n\n# To get the results, e.g., predictions_df = linker.inference.predict()","lang":"python","description":"This quickstart demonstrates how to set up a basic Splink deduplication model using DuckDB. It covers defining comparison libraries and blocking rules, estimating parameters for record linkage, and preparing for prediction. It uses a built-in `fake_1000` dataset for convenience."},"warnings":[{"fix":"Review v5.0 documentation and migration guides for updated API calls, cache management, and probabilistic calculation handling. Users relying on Athena should plan for migration to another backend or use Splink v4.x.","message":"Splink v5.0 introduces significant breaking changes. Key updates include the removal of the implicit cache mechanism in favor of explicit cache table management functions, removal of 'salting', introduction of 'chunking' for large datasets, and a shift from Bayes Factors to Match Weights (log-odds) for internal probabilistic calculations to improve numerical stability. Additionally, support for the Athena backend is being dropped.","severity":"breaking","affected_versions":">=5.0.0"},{"fix":"Ensure your environment uses Python 3.9 or higher. The current requirement is `>=3.9.0, <4.0.0`.","message":"Python 3.8 support was dropped in Splink v4.0.12. Older versions of Python are being phased out in alignment with community end-of-life policies.","severity":"breaking","affected_versions":"<4.0.12 (Python 3.8)"},{"fix":"Pre-process data to ensure multiple, diverse columns are used for linkage. Avoid relying on highly correlated features or single 'bag of words' columns for optimal accuracy.","message":"Splink performs best with input data containing multiple, non-highly correlated columns. It is not designed for linking single-column 'bag of words' data (e.g., only a company name). High correlation (e.g., city and postcode) can also reduce effectiveness.","severity":"gotcha","affected_versions":"All"},{"fix":"For optimal performance and feature coverage, especially with larger datasets or complex comparisons, consider using DuckDB (default) or other actively supported backends like Spark or PostgreSQL.","message":"SQLite backend support is minimal and receives less attention from the development team compared to DuckDB and Spark. It has reasonable but not complete coverage of comparison functions, particularly for array and date comparisons.","severity":"deprecated","affected_versions":"All"}],"env_vars":null,"search_vec":"'100':60 'accur':9 'allow':53 'apach':48 'athena':52 'aw':51 'backend':45 'base':34 'big':84 'data':77,85,87 'dataset':26,58 'dedupl':21,81 'diagnost':76 'duckdb':47 'enabl':18 'entiti':15,79 'fast':8 'fellegi':38 'fellegi-sunt':37 'identifi':30 'interact':70 'lack':28 'learn':33 'leverag':31 'like':46 'link':23 'linkag':14,78 'match':83 'million':61 'model':40,73 'packag':6 'probabilist':12,82 'provid':66 'python':5 'record':13,24,62 'resolut':16,80 'scalabl':11 'scale':56 'scienc':88 'spark':49 'splink':1,2,41 'sql':44,86 'suit':68 'sunter':39 'support':42 'understand':74 'uniqu':29 'unsupervis':32 'user':19 'various':43 'visual':71","created_at":"2026-04-14T01:36:29.195989+00:00","updated_at":"2026-04-16T22:09:04.240257+00:00","problems":[{"fix":"Install Splink using pip, specifying your desired SQL backend (e.g., '[duckdb]', '[spark]').","cause":"The 'splink' package is not installed in the current Python environment or virtual environment.","error":"ModuleNotFoundError: No module named 'splink'"},{"fix":"Use `linker.get_pairwise_matches()` to generate all potential matches with their probabilities, or `linker.get_ranked_probas()` for a ranked list of record pairs. Example: `matches = linker.get_pairwise_matches()`","cause":"In Splink v4.0+, the `predict` method was replaced or renamed with more granular methods like `get_pairwise_matches()` or `get_ranked_probas()` for improved functionality.","error":"AttributeError: 'Linker' object has no attribute 'predict'"},{"fix":"Update your code to use the new API. For example, replace `splink.analyse.estimate_u_values('col')` with `linker.estimate_u_values('col')` or `splink.analyse.estimate_probability_two_random_records_match` with `linker.compute_blocking_rule_performance()`.","cause":"The `splink.analyse` module was removed in Splink v4.0. Its functions were integrated directly into the `Linker` object or replaced by new methods.","error":"AttributeError: module 'splink' has no attribute 'analyse'"},{"fix":"Ensure your data is registered with the `Linker` using `linker.register_table(your_dataframe, 'your_table_name')` and that the table name in your settings dictionary or queries exactly matches the registered name.","cause":"Splink's underlying SQL engine (e.g., DuckDB) cannot find the specified table. This often happens if the data frame or table was not correctly registered with the `Linker` object or if there's a typo in the table name.","error":"duckdb.DuckDBError: Table with name '...' does not exist!"}],"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"4.0.16","cli_name":"","cli_version":null,"type":"library","homepage":"https://moj-analytical-services.github.io/splink/","github":"https://github.com/moj-analytical-services/splink","docs":null,"changelog":null,"pypi":"https://pypi.org/project/splink/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","database","ai-ml"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-30","next_check":"2026-07-28","install_tag":null}}