pipeline.helpers
retry_load
def retry_load(retry_on_pipeline_steps: Sequence[TPipelineStep] = (
"load", )) -> Callable[[BaseException], bool]
A retry strategy for Tenacity that, with default setting, will repeat load
step for all exceptions that are not terminal
Use this condition with tenacity retry_if_exception
. Terminal exceptions are exceptions that will not go away when operations is repeated.
Examples: missing configuration values, Authentication Errors, terminally failed jobs exceptions etc.
data = source(...)
for attempt in Retrying(stop=stop_after_attempt(3), retry=retry_if_exception(retry_load(())), reraise=True):
with attempt:
p.run(data)
Arguments:
retry_on_pipeline_steps
Tuple[TPipelineStep, ...], optional - which pipeline steps are allowed to be repeated. Default: "load"
DropCommand Objects
class DropCommand()
__init__
def __init__(pipeline: "Pipeline",
resources: Union[Iterable[Union[str, TSimpleRegex]],
Union[str, TSimpleRegex]] = (),
schema_name: Optional[str] = None,
state_paths: TAnyJsonPath = (),
drop_all: bool = False,
state_only: bool = False) -> None
Arguments:
pipeline
- Pipeline to drop tables and state fromresources
- List of resources to drop. If empty, no resources are dropped unlessdrop_all
is Trueschema_name
- Name of the schema to drop tables from. If not specified, the default schema is usedstate_paths
- JSON path(s) relative to the source state to dropdrop_all
- Drop all resources and tables in the schema (supersedesresources
list)state_only
- Drop only state, not tables
refresh_source
def refresh_source(pipeline: "Pipeline", source: DltSource,
refresh: TRefreshMode) -> TLoadPackageDropTablesState
Run the pipeline's refresh mode on the given source, updating the provided schema
and pipeline state.
Returns:
The new load package state containing tables that need to be dropped/truncated.