extract.incremental.transform
IncrementalTransform Objects
class IncrementalTransform()
A base class for handling extraction and stateful tracking of incremental data from input data items.
By default, the descendant classes are instantiated within the
dlt.extract.incremental.Incremental
class.
Subclasses must implement the __call__
method which will be called
for each data item in the extracted data.
deduplication_disabled
@property
def deduplication_disabled() -> bool
Skip deduplication when length of the key is 0
JsonIncremental Objects
class JsonIncremental(IncrementalTransform)
Extracts incremental data from JSON data items.
find_cursor_value
def find_cursor_value(row: TDataItem) -> Any
Finds value in row at cursor defined by self.cursor_path.
Will use compiled JSONPath if present, otherwise it reverts to column search if row is dict
__call__
def __call__(row: TDataItem) -> Tuple[Optional[TDataItem], bool, bool]
Returns:
Tuple (row, start_out_of_range, end_out_of_range) where row is either the data item or None
if it is completely filtered out