Skip to main content
Version: devel

dlt.common.libs.pyarrow

UnsupportedArrowTypeException Objects​

class UnsupportedArrowTypeException(DltException)

View source on GitHub

Exception raised when Arrow type conversion failed.

The setters are used to update the exception with more context such as the relevant field and tablea it is caught downstream.

PyToArrowConversionException Objects​

class PyToArrowConversionException(DltException)

View source on GitHub

Exception raised when converting data to Arrow based on a TableSchema

ArrowSchemaNormalizationResult Objects​

class ArrowSchemaNormalizationResult(NamedTuple)

View source on GitHub

Named result of should_normalize_arrow_schema

Fields:

  • should_normalize: whether any normalization is required
  • rename_mapping: mapping from original field names to normalized names
  • rev_mapping: reverse mapping from normalized names to original
  • nullable_updates: fields that require nullable flag updates (by normalized name)
  • columns: potentially filtered/adjusted TTableSchemaColumns

get_column_type_from_py_arrow​

def get_column_type_from_py_arrow(dtype: pyarrow.DataType) -> TColumnType

View source on GitHub

Returns (data_type, precision, scale) tuple from pyarrow.DataType

py_arrow_to_table_schema_columns​

def py_arrow_to_table_schema_columns(
        schema: pyarrow.Schema) -> TTableSchemaColumns

View source on GitHub

Convert a PyArrow schema to a table schema columns dict.

Arguments:

  • schema pyarrow.Schema - pyarrow schema

Returns:

  • TTableSchemaColumns - table schema columns

get_nested_column_type_from_py_arrow​

def get_nested_column_type_from_py_arrow(
        dtype: pyarrow.DataType) -> TColumnType

View source on GitHub

Creates json dlt data type with nested type structure in x-nested-type hint. Currently the only recognized nested type format is arrow-ipc

serialize_type​

def serialize_type(dtype: pyarrow.DataType) -> str

View source on GitHub

Serializes arrow type via arrow ipc as base64 str

remove_null_columns​

def remove_null_columns(item: TAnyArrowItem) -> TAnyArrowItem

View source on GitHub

Remove all columns of datatype pyarrow.null() from the table or record batch. Stores removed column names in arrow schema metadata under 'dlt.null_columns' key.

remove_null_columns_from_schema​

def remove_null_columns_from_schema(
        schema: pyarrow.Schema) -> Tuple[pyarrow.Schema, bool]

View source on GitHub

Remove all columns of datatype pyarrow.null() from the schema

remove_columns​

def remove_columns(item: TAnyArrowItem,
                   columns: Sequence[str]) -> TAnyArrowItem

View source on GitHub

Remove columns from Arrow item

append_column​

def append_column(item: TAnyArrowItem, name: str, data: Any) -> TAnyArrowItem

View source on GitHub

Appends new column to Table or RecordBatch

rename_columns​

def rename_columns(item: TAnyArrowItem,
                   new_column_names: Sequence[str]) -> TAnyArrowItem

View source on GitHub

Rename arrow columns on Table or RecordBatch, returns same data but with renamed schema

fill_empty_source_column_values_with_placeholder​

def fill_empty_source_column_values_with_placeholder(
        table: pyarrow.Table, source_columns: List[str],
        placeholder: str) -> pyarrow.Table

View source on GitHub

Replaces empty strings and null values in the specified source columns of an Arrow table with a placeholder string.

Arguments:

  • table pa.Table - The input Arrow table.
  • source_columns List[str] - A list of column names to replace empty strings and null values in.
  • placeholder str - The placeholder string to use for replacement.

Returns:

  • pyarrow.Table - The modified Arrow table with empty strings and null values replaced in the specified columns.

should_normalize_arrow_schema​

def should_normalize_arrow_schema(
        schema: pyarrow.Schema, columns: TTableSchemaColumns,
        naming: NamingConvention) -> ArrowSchemaNormalizationResult

View source on GitHub

Figure out if any of the normalization steps must be executed. This prevents from rewriting arrow tables when no changes are needed. Refer to normalize_py_arrow_item for a list of normalizations. Note that column must be already normalized.

normalize_py_arrow_item​

def normalize_py_arrow_item(
        item: TAnyArrowItem, columns: TTableSchemaColumns,
        naming: NamingConvention,
        caps: DestinationCapabilitiesContext) -> TAnyArrowItem

View source on GitHub

Normalize arrow item schema according to the columns. Note that columns must be already normalized.

  1. columns with no data type will be dropped
  2. arrow schema field names will be normalized according to naming
  3. arrows columns will be reordered according to columns
  4. empty columns will be inserted if they are missing, types will be generated using caps
  5. arrow columns with different nullability than corresponding schema columns will be updated
  6. timestamps will be normalized according to timezone flag in dlt column schema

NOTE: nullability is not enforced. it is up to destination to do that.

normalize_py_arrow_item_column​

def normalize_py_arrow_item_column(
        column: TColumnSchema, arrow_type: pyarrow.Field,
        arrow_column: pyarrow.Array) -> Tuple[pyarrow.DataType, pyarrow.Array]

View source on GitHub

Normalize arrow timestamp column timezone according to dlt schema column convention.

Arguments:

  • column - dlt column schema with timezone hint
  • arrow_type - actual PyArrow data type
  • arrow_column - actual PyArrow column data

Returns:

Tuple of (modified_type, modified_column) or (arrow_field, arrow_column) if no changes needed

add_dlt_load_id_column​

def add_dlt_load_id_column(item: TAnyArrowItem, columns: TTableSchemaColumns,
                           caps: DestinationCapabilitiesContext,
                           naming: NamingConvention,
                           load_id: str) -> TAnyArrowItem

View source on GitHub

Adds or replaces the _dlt_load_id column.

get_normalized_arrow_fields_mapping​

def get_normalized_arrow_fields_mapping(schema: pyarrow.Schema,
                                        naming: NamingConvention) -> StrStr

View source on GitHub

Normalizes schema field names and returns mapping from original to normalized name. Raises on name collisions

dlt_column_to_arrow_field​

def dlt_column_to_arrow_field(
        column: TColumnSchema,
        caps: DestinationCapabilitiesContext,
        timestamp_timezone: str = "UTC") -> pyarrow.Field

View source on GitHub

Convert a single dlt column schema to a PyArrow field.

Arguments:

  • column TColumnSchema - dlt column schema with at least name and data_type.
  • caps DestinationCapabilitiesContext - Destination capabilities for type mapping.
  • timestamp_timezone str - Timezone for timestamp columns.

Returns:

  • pyarrow.Field - Corresponding PyArrow field.

columns_to_arrow​

def columns_to_arrow(columns: TTableSchemaColumns,
                     caps: DestinationCapabilitiesContext,
                     timestamp_timezone: str = "UTC") -> pyarrow.Schema

View source on GitHub

Convert a table schema columns dict to a pyarrow schema.

Arguments:

  • columns TTableSchemaColumns - table schema columns

Returns:

  • pyarrow.Schema - pyarrow schema

get_parquet_metadata​

def get_parquet_metadata(
        parquet_file: TFileOrPath) -> Tuple[int, pyarrow.Schema]

View source on GitHub

Gets parquet file metadata (including row count and schema)

Arguments:

  • parquet_file str - path to parquet file

Returns:

  • FileMetaData - file metadata

to_arrow_scalar​

def to_arrow_scalar(value: Any, arrow_type: pyarrow.DataType) -> Any

View source on GitHub

Converts python value to an arrow compute friendly version

from_arrow_scalar​

def from_arrow_scalar(arrow_value: pyarrow.Scalar) -> Any

View source on GitHub

Converts arrow scalar into Python type.

TNewColumns​

Sequence of tuples: (field index, field, generating function)

add_constant_column​

def add_constant_column(item: TAnyArrowItem,
                        name: str,
                        data_type: pyarrow.DataType,
                        value: Any = None,
                        nullable: bool = True,
                        index: int = -1,
                        use_dictionary: bool = True) -> TAnyArrowItem

View source on GitHub

Add column with a single value to the table.

Arguments:

  • item - Arrow table or record batch
  • name - The new column name
  • data_type - The data type of the new column
  • nullable - Whether the new column is nullable
  • value - The value to fill the new column with
  • index - The index at which to insert the new column. Defaults to -1 (append)
  • use_dictionary - When True (default), creates a dictionary-encoded column which is memory-efficient for repeated values. Set to False for destinations that don't support dictionary types (e.g., ADBC drivers for MSSQL).

Notes:

When use_dictionary=True, the column is created as a DictionaryArray with int8 indices. When use_dictionary=False, a regular array filled with the repeated value is created.

pq_stream_with_new_columns​

def pq_stream_with_new_columns(
        parquet_file: TFileOrPath,
        columns: TNewColumns,
        row_groups_per_read: int = 1) -> Iterator[pyarrow.Table]

View source on GitHub

Add column(s) to the table in batches.

The table is read from parquet row_groups_per_read row groups at a time

Arguments:

  • parquet_file - path or file object to parquet file
  • columns - list of columns to add in the form of (insertion index, pyarrow.Field, column_value_callback) The callback should accept a pyarrow.Table and return an array of values for the column.
  • row_groups_per_read - number of row groups to read at a time. Defaults to 1.

Yields:

pyarrow.Table objects with the new columns added.

cast_arrow_schema_types​

def cast_arrow_schema_types(
    schema: pyarrow.Schema, type_map: Dict[Callable[[pyarrow.DataType], bool],
                                           Callable[..., pyarrow.DataType]]
) -> pyarrow.Schema

View source on GitHub

Returns type-casted Arrow schema.

Replaces data types for fields matching a type check in type_map. Type check functions in type_map are assumed to be mutually exclusive, i.e. a data type does not match more than one type check function.

concat_batches_and_tables_in_order​

def concat_batches_and_tables_in_order(
        tables_or_batches: Iterable[Union[pyarrow.Table, pyarrow.RecordBatch]],
        promote_options: str = "none") -> pyarrow.Table

View source on GitHub

Concatenate iterable of tables and batches into a single table, preserving row order.

Arguments:

  • promote_options - PyArrow concat_tables promote_options. "none" (default) requires identical schemas and enables zero-copy concat. "default" promotes within type families (e.g. int32→int64). "permissive" promotes across families (e.g. int64→double).

transpose_rows_to_columns​

def transpose_rows_to_columns(rows: TDataItems,
                              column_names: Iterable[str]) -> dict[str, Any]

View source on GitHub

Transpose rows (data items) into columns (numpy arrays). Returns a dictionary of {column_name: column_data}

Uses pandas if available. Otherwise, use numpy, which is slower

uuid_to_string​

def uuid_to_string(arr: Any) -> Any

View source on GitHub

Convert an array of UUIDs to canonical hyphenated lowercase string array.

Accepts a pa.uuid() extension array or a fixed_size_binary[16]. Nulls are preserved. Sliced inputs (storage.offset != 0) are supported. Falls back to pure Python if numpy is missing or the input is sliced.

convert_numpy_to_arrow​

def convert_numpy_to_arrow(column_data: Any,
                           caps: DestinationCapabilitiesContext,
                           column_schema: TColumnSchema, tz: str,
                           safe_arrow_conversion: bool) -> Any

View source on GitHub

Convert a numpy array to a pyarrow array.

Arguments:

Returns:

an arrow Array

cast_arrow_as_columns_schema​

def cast_arrow_as_columns_schema(
        item: TAnyArrowItem,
        columns: TTableSchemaColumns,
        caps: DestinationCapabilitiesContext,
        tz: str,
        safe_arrow_conversion: bool = True) -> TAnyArrowItem

View source on GitHub

Cast an Arrow item to match dlt column schema.

Performs per-column casts using Arrow, with fallbacks equivalent to cast_arrow_array_as_column_schema. Timestamp timezone normalization is applied when required by the dlt schema. The result preserves the input item's schema metadata and per-field metadata.

Arguments:

  • item - Arrow table or record batch to cast.
  • columns - dlt table schema columns containing data type hints.
  • caps - capabilities of the storage backend.
  • tz - time zone identifier used to resolve timestamp hints.
  • safe_arrow_conversion - if False, truncation and loss of precision is allowed.

Returns:

The same kind of Arrow object (Table or RecordBatch) with columns cast to the requested dlt types. Schema and field metadata are preserved.

row_tuples_to_arrow​

def row_tuples_to_arrow(rows: TDataItems,
                        caps: DestinationCapabilitiesContext,
                        columns: TTableSchemaColumns,
                        tz: str,
                        safe_arrow_conversion: bool = True) -> Any

View source on GitHub

Converts the rows to an arrow table using the columns schema.

  1. Pivot rows into columns.
  2. Convert columns to pyarrow arrays; coerce type and nullability; exclude types unsupported by arrow
  3. Create table
  4. Remove columns full of null values

Arguments:

Returns:

an arrow Table

set_plus0000_timezone_to_utc​

def set_plus0000_timezone_to_utc(tbl: pyarrow.Table) -> pyarrow.Table

View source on GitHub

Convert any +00:00 timestamp columns to UTC. Returns the original table object if nothing needed fixing.

cast_date64_columns_to_timestamp​

def cast_date64_columns_to_timestamp(tbl: pyarrow.Table,
                                     tz: Optional[str] = None
                                     ) -> pyarrow.Table

View source on GitHub

Cast any date64 columns to timestamp with microsecond precision, preserving the semantic time values. Uses pyarrow.compute.cast on the column (works for chunked arrays) to cast from milliseconds (date64) to microseconds (timestamp[us]).

Arguments:

  • tbl - Input Arrow table.
  • tz - Optional timezone to annotate the resulting timestamp with (e.g. "UTC"). If None (default), produces a naive timestamp.

Returns:

A new table with date64 columns cast to timestamp[us] (optionally tz-aware), or the original table if no date64 columns were found.

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.