extract.incremental.transform

IncrementalTransform Objects

class IncrementalTransform()

[view_source]

A base class for handling extraction and stateful tracking of incremental data from input data items.

By default, the descendant classes are instantiated within the dlt.extract.incremental.Incremental class.

Subclasses must implement the __call__ method which will be called for each data item in the extracted data.

deduplication_disabled

@property
def deduplication_disabled() -> bool

[view_source]

Skip deduplication when length of the key is 0 or if lag is applied.

JsonIncremental Objects

class JsonIncremental(IncrementalTransform)

[view_source]

Extracts incremental data from JSON data items.

find_cursor_value

def find_cursor_value(row: TDataItem) -> Any

[view_source]

Finds value in row at cursor defined by self.cursor_path.

Will use compiled JSONPath if present. Otherwise, reverts to field access if row is dict, Pydantic model, or of other class.

__call__

def __call__(row: TDataItem) -> Tuple[Optional[TDataItem], bool, bool]

[view_source]

Returns:

Tuple (row, start_out_of_range, end_out_of_range) where row is either the data item or None if it is completely filtered out