pipeline.configuration
PipelineConfiguration Objects
@configspec
class PipelineConfiguration(BaseConfiguration)
dataset_name_layout
Layout for datasetname, where %s is replaced with dataset_name. For example: 'prefix%s'
restore_from_destination
Enables the run
method of the Pipeline
object to restore the pipeline state and schemas from the destination
enable_runtime_trace
Enables the tracing. Tracing saves the execution trace locally and is required by dlt deploy
.
use_single_dataset
Stores all schemas in single dataset. When False, each schema will get a separate dataset with `{datasetname}{schema_name}
full_refresh
Deprecated. Use dev_mode
instead. When set to True, each instance of the pipeline with the pipeline_name
starts from scratch when run and loads the data to a separate dataset.
dev_mode
When set to True, each instance of the pipeline with the pipeline_name
starts from scratch when run and loads the data to a separate dataset.
refresh
Refresh mode for the pipeline to fully or partially reset a source during run. See docstring of dlt.pipeline
for more details.