{"id":7813,"library":"typedspark","title":"Column-wise type annotations for pyspark DataFrames","description":"Typedspark provides column-wise type annotations for PySpark DataFrames, enhancing code readability, enabling static type-checking and linting, and improving auto-completion and refactoring capabilities. It helps define explicit schemas for Spark DataFrames, ensuring data integrity at a structural level. The library is currently at version 1.6.3 and maintains a regular release cadence, often driven by dependency updates.","status":"active","version":"1.6.3","language":"python","source_language":"en","source_url":"https://github.com/kaiko-ai/typedspark","tags":["pyspark","spark","typing","type-checking","data-quality","etl","schema-validation"],"install":[{"cmd":"pip install typedspark","lang":"bash","label":"Basic Installation"},{"cmd":"pip install \"typedspark[pyspark]\"","lang":"bash","label":"Installation with PySpark (if not pre-installed)"}],"dependencies":[{"reason":"Core dependency for PySpark DataFrame functionality; optional during installation for environments with pre-installed PySpark (e.g., Databricks, EMR).","package":"pyspark","optional":true}],"imports":[{"wrong":"from typedspark import Column","symbol":"Column","correct":"from typedspark import Column"}],"quickstart":{"code":"import pyspark.sql.functions as F\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.types import LongType, StringType\nfrom typedspark import Column, DataSet, Schema\n\n# Initialize Spark Session (if not already present)\nspark = SparkSession.builder.appName(\"TypedSparkQuickstart\").getOrCreate()\n\nclass Person(Schema):\n    id: Column[LongType]\n    name: Column[StringType]\n    age: Column[LongType]\n\ndef process_person_data(df: DataSet[Person]) -> DataSet[Person]:\n    # Example transformation: add 1 to age\n    return df.withColumn(Person.age, F.col(Person.age) + 1)\n\n# Create a dummy DataFrame conforming to the Person schema\ndata = [\n    (1, \"Alice\", 30),\n    (2, \"Bob\", 24),\n    (3, \"Charlie\", 35)\n]\nschema_spark = Person.get_structtype()\ndf_untyped = spark.createDataFrame(data, schema=schema_spark)\n\n# Convert to a TypedSpark DataSet\ndf_typed = DataSet[Person](df_untyped)\n\n# Process the data using the typed function\ndf_processed = process_person_data(df_typed)\n\n# Show results\ndf_processed.show()\n\n# You can also generate an empty DataSet for testing\nempty_person_dataset = Person.create_empty_dataset(spark)\nempty_person_dataset.show()\n\nspark.stop()","lang":"python","description":"This quickstart demonstrates how to define a schema using `typedspark.Schema`, create a `DataSet` from a PySpark DataFrame, and apply transformations with type annotations. It also shows how to create an empty `DataSet` for testing purposes."},"warnings":[{"fix":"Refer to the `typedspark` documentation or GitHub README for the officially supported PySpark versions for your `typedspark` release. Pin your `pyspark` dependency accordingly.","message":"Typedspark's compatibility is tested with specific PySpark versions (e.g., 3.5.7 and 4.1.0 for v1.6.3). Using untested or significantly different PySpark versions may lead to unexpected behavior or incompatibilities.","severity":"gotcha","affected_versions":"<1.6.3 (for newer PySpark versions)"},{"fix":"Upgrade to `typedspark` version 1.6.2 or newer to ensure correct `Column` comparison behavior in threaded contexts. If upgrading is not possible, ensure Spark operations are not performed in parallel threads where `typedspark.Column` instances are compared.","message":"Prior to version 1.6.2, `Column` comparison in multi-threaded environments could lead to issues. This was fixed by explicitly using `SparkSession.active()` for thread-safe operations.","severity":"breaking","affected_versions":"<1.6.2"},{"fix":"Implement runtime validation mechanisms if strict schema enforcement is required at various pipeline stages. `typedspark` itself provides methods for validation, but these need to be explicitly invoked in your data pipeline after untyped operations or external data reads.","message":"While `typedspark` provides compile-time type-checking, runtime schema mismatches can still occur if the underlying PySpark DataFrame's schema changes unexpectedly after a `DataSet` is created (e.g., due to an external data source modification or an untyped transformation).","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"search_vec":"'1.6.3':58 'annot':5,15 'auto':32 'auto-complet':31 'cadenc':64 'capabl':36 'check':26,75 'code':20 'column':2,12 'column-wis':1,11 'complet':33 'current':55 'data':46,77 'data-qu':76 'datafram':8,18,44 'defin':39 'depend':68 'driven':66 'enabl':22 'enhanc':19 'ensur':45 'etl':79 'explicit':40 'help':38 'improv':30 'integr':47 'level':51 'librari':53 'lint':28 'maintain':60 'often':65 'provid':10 'pyspark':7,17,70 'qualiti':78 'readabl':21 'refactor':35 'regular':62 'releas':63 'schema':41,81 'schema-valid':80 'spark':43,71 'static':23 'structur':50 'type':4,14,25,72,74 'type-check':24,73 'typedspark':9 'updat':69 'valid':82 'version':57 'wise':3,13","created_at":"2026-04-16T14:14:29.749660+00:00","updated_at":"2026-04-16T14:14:29.749660+00:00","problems":{"verify_error":"Traceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"/tmp/tmp9n8o9yyn/venv/lib/python3.12/site-packages/typedspark/__init__.py\", line 3, in <module>\n    from typedspark._core.column import Column\n  File \"/tmp/tmp9n8o9yyn/venv/lib/python3.12/site-packages/typedspark/_core"},"ecosystem":"pypi","meta_description":null,"install_score":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":"1.7.0","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/kaiko-ai/typedspark","docs":null,"changelog":null,"pypi":"https://pypi.org/project/typedspark/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["data","database","type-stubs"],"base_url":null,"auth_type":null,"provenance":{"verified_status":"import_fail","verified_at":"2026-07-03","last_verified":"2026-07-03","next_check":"2026-07-10","install_tag":null}}