{"id":3286,"library":"sudachipy","title":"SudachiPy","description":"SudachiPy is a Python binding for Sudachi.rs, a Japanese morphological analyzer implemented in Rust. It provides multi-granular tokenization for Japanese text, handling various linguistic nuances. The current version is 0.6.10, with releases typically occurring every few months to incorporate updates, bug fixes, and Python version support.","status":"active","version":"0.6.10","language":"python","source_language":"en","source_url":"https://github.com/WorksApplications/sudachi.rs/tree/develop/python","tags":["NLP","Japanese","morphological analysis","tokenization","text processing"],"install":[{"cmd":"pip install sudachipy sudachidict_core","lang":"bash","label":"Install SudachiPy and Core Dictionary"}],"dependencies":[{"reason":"Required for dictionary functionality; not bundled with sudachipy.","package":"sudachidict_core","optional":false},{"reason":"Required for building from source, particularly on ARM architectures.","package":"setuptools-rust","optional":true},{"reason":"Rust compiler toolchain required for building from source (e.g., on ARM64 Linux/macOS).","package":"rust","optional":true}],"imports":[{"note":"Importing from sudachipy.dictionary is deprecated as of recent versions.","wrong":"from sudachipy.dictionary import Dictionary","symbol":"Dictionary","correct":"from sudachipy import Dictionary"},{"note":"Importing from sudachipy.tokenizer is deprecated as of recent versions.","wrong":"from sudachipy.tokenizer import Tokenizer","symbol":"Tokenizer","correct":"from sudachipy import Tokenizer"},{"symbol":"SplitMode","correct":"from sudachipy import SplitMode"}],"quickstart":{"code":"from sudachipy import Dictionary, SplitMode\n\n# Initialize the tokenizer with the default (core) dictionary\ntokenizer = Dictionary().create()\n\ntext = \"すもももももももものうち\"\n\n# Tokenize in SplitMode.C (shortest segmentation)\nmorphemes_c = tokenizer.tokenize(text, SplitMode.C)\nprint(\"SplitMode.C:\", [m.surface() for m in morphemes_c])\n\n# Tokenize in SplitMode.A (medium segmentation)\nmorphemes_a = tokenizer.tokenize(text, SplitMode.A)\nprint(\"SplitMode.A:\", [m.surface() for m in morphemes_a])\n\n# Access morpheme details\nif morphemes_c:\n    first_morpheme = morphemes_c[0]\n    print(f\"\\nFirst morpheme (C): {first_morpheme.surface()}\")\n    print(f\"  Reading form: {first_morpheme.reading_form()}\")\n    print(f\"  Part of Speech: {first_morpheme.part_of_speech()}\")","lang":"python","description":"This quickstart demonstrates how to initialize the SudachiPy tokenizer with a default dictionary and perform multi-granular tokenization on Japanese text. It also shows how to access basic information for individual morphemes."},"warnings":[{"fix":"Upgrade your Python environment to 3.9 or a later supported version.","message":"Support for Python 3.6, 3.7, and 3.8 has been removed in versions 0.6.4 and 0.6.9, respectively. Ensure you are using Python 3.9 or newer.","severity":"breaking","affected_versions":">=0.6.4, >=0.6.9"},{"fix":"Refer to the documentation for updated methods of specifying dictionary paths or types (e.g., `Dictionary(dict_type='full')`).","message":"The `sudachipy link` command, used for managing dictionary paths, was removed in v0.5.2 and later. Dictionary specification methods now rely on `config_path` or `dict_type` arguments to `Dictionary()` or CLI options.","severity":"breaking","affected_versions":">=0.5.2"},{"fix":"Change import statements to `from sudachipy import Dictionary, Tokenizer, SplitMode`.","message":"Direct imports like `from sudachipy.dictionary import Dictionary` and `from sudachipy.tokenizer import Tokenizer` are deprecated. Import `Dictionary`, `Tokenizer`, and `SplitMode` directly from the top-level `sudachipy` package.","severity":"deprecated","affected_versions":">=0.6.x"},{"fix":"Always install a dictionary package alongside `sudachipy`, e.g., `pip install sudachipy sudachidict_core`.","message":"SudachiPy requires a dictionary package (e.g., `sudachidict_core`) to be installed separately. It is not included in the main `sudachipy` package.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Install Rust and `setuptools-rust` (e.g., `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && pip install setuptools-rust`).","message":"Building SudachiPy from source (common on ARM64 Linux/macOS if no pre-built wheel is available) requires the Rust compiler toolchain and `setuptools-rust` to be installed in your environment.","severity":"gotcha","affected_versions":"All versions (when building from source)"},{"fix":"Create different `Tokenizer` instances for different modes, e.g., `tokenizer_a = Dictionary().create(mode=SplitMode.A)`.","message":"The `mode` parameter in the `Tokenizer.tokenize()` method is deprecated. Pass the analysis mode when creating the `Tokenizer` instance, or use `Morpheme.split()` for multi-level splitting.","severity":"gotcha","affected_versions":">=0.6.x"},{"fix":"If custom dictionary paths are used, verify they align with the new resolution order. Consider using the `resource_dir` parameter in `Dictionary()` constructor or `config_path` for explicit control.","message":"Dictionary resource path resolution logic changed in v0.6.3. Paths are now resolved in a specific order: absolute paths, relative to config `path`, relative to `resource_dir` param, relative to config file, relative to current directory.","severity":"gotcha","affected_versions":">=0.6.3"}],"env_vars":null,"search_vec":"'0.6.10':33 'analysi':53 'analyz':12 'bind':6 'bug':44 'current':30 'everi':38 'fix':45 'granular':20 'handl':25 'implement':13 'incorpor':42 'japanes':10,23,51 'linguist':27 'month':40 'morpholog':11,52 'multi':19 'multi-granular':18 'nlp':50 'nuanc':28 'occur':37 'process':56 'provid':17 'python':5,47 'releas':35 'rust':15 'sudachi.rs':8 'sudachipi':1,2 'support':49 'text':24,55 'token':21,54 'typic':36 'updat':43 'various':26 'version':31,48","created_at":"2026-04-11T09:28:59.958542+00:00","updated_at":"2026-04-16T22:31:49.683712+00:00","problems":{"verify_error":"error: Failed to parse: `sudachipy sudachidict_core`\n  Caused by: Expected one of `@`, `(`, `<`, `=`, `>`, `~`, `!`, `;`, found `s`\nsudachipy sudachidict_core\n          ^"},"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"0.6.11","cli_name":"","cli_version":null,"type":"library","homepage":"https://worksapplications.github.io/sudachi.rs/python/","github":"https://github.com/WorksApplications/sudachi.rs","docs":null,"changelog":null,"pypi":"https://pypi.org/project/sudachipy/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["ai-ml","data"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"passing","verified_at":"2026-06-28","last_verified":"2026-08-29","next_check":"2026-07-05","install_tag":null}}