dlt.sources.helpers.transform

take_first

def take_first(max_items: int) -> ItemTransformFunctionNoMeta[bool]

View source on GitHub

A filter that takes only first max_items from a resource

skip_first

def skip_first(max_items: int) -> ItemTransformFunctionNoMeta[bool]

View source on GitHub

A filter that skips first max_items from a resource

pivot

def pivot(paths: Union[str, Sequence[str]] = "$",
          prefix: str = "col") -> ItemTransformFunctionNoMeta[TDataItem]

View source on GitHub

Pivot the given sequence of sequences into a sequence of dicts, generating column names from the given prefix and indexes, e.g.: {"field": [[1, 2]]} -> {"field": [{"prefix_0": 1, "prefix_1": 2}]}

Arguments:

• paths Union[str, Sequence[str]] - JSON paths of the fields to pivot.
• prefix Optional[str] - Prefix to add to the column names.

Returns:

• ItemTransformFunctionNoMeta[TDataItem] - The transformer function.

propagate_to_nested

def propagate_to_nested(
    keys: Union[str, Sequence[str]],
    max_nesting: Optional[int] = None
) -> ItemTransformFunctionNoMeta[TDataItem]

View source on GitHub

Copies the values of keys of a root item into the dicts of its nested lists, so the rows of nested tables get the values of their root row, e.g.: {"day": 1, "lines": [{"sku": "a"}]} -> {"day": 1, "lines": [{"sku": "a", "day": 1}]}

The copied values replace the values of the nested dicts. Dicts nested in dicts are flattened into the same table, so they do not get the values. Changes the item in place, items that are not dicts pass through.

Arguments:

• keys Union[str, Sequence[str]] - Keys of the root item to copy.
• max_nesting Optional[int] - The number of nested list levels that get the values, all levels if None.

Returns:

• ItemTransformFunctionNoMeta[TDataItem] - The map function for resource.add_map.

add_row_hash_to_table

def add_row_hash_to_table(row_hash_column_name: str) -> TDataItem

View source on GitHub

Computes content hash for each row of pandas DataFrame, arrow table or batch and adds it as row_hash_column_name column.

Internally arrow tables and batches are converted to pandas DataFrame and then hash_pandas_object is used to generate a series with row hashes. Hashes are converted to signed int64 and added to original table. Data may be modified. For SCD2 use with a resource configuration that assigns custom row version column to row_hash_column_name