dlt.extract.incremental.transform
IncrementalTransform Objects
class IncrementalTransform()
A base class for handling extraction and stateful tracking of incremental data from input data items.
By default, the descendant classes are instantiated within the
dlt.extract.incremental.Incremental class.
Subclasses must implement the __call__ method which will be called
for each data item in the extracted data.
deduplication_disabled
@property
def deduplication_disabled() -> bool
Skip deduplication when length of the key is 0 or if lag is applied.
JsonIncremental Objects
class JsonIncremental(IncrementalTransform)
Extracts incremental data from JSON data items.
find_cursor_value
def find_cursor_value(row: TDataItem) -> Any
Finds value in row at cursor defined by self.cursor_path.
Will use compiled JSONPath if present. Otherwise, reverts to field access if row is dict, Pydantic model, or of other class.
__call__
def __call__(row: TDataItem) -> Tuple[Optional[TDataItem], bool, bool]
Returns:
Tuple (row, start_out_of_range, end_out_of_range) where row is either the data item or None if it is completely filtered out