dlt.common.metrics
aggregate_job_metrics
def aggregate_job_metrics(
job_metrics: Mapping[TJobKey, DataWriterMetrics],
key: Callable[[TJobKey], str]) -> Dict[str, DataWriterMetrics]
Sum writer metrics grouped by key, independent of input order.
StepMetrics Objects
class StepMetrics(TypedDict)
Metrics for particular package processed in particular pipeline step
started_at
Start of package processing
finished_at
End of package processing
TDataLocation Objects
class TDataLocation(TypedDict)
A logical data source or target: a dataset of tables, an API of endpoints, a bucket of files.
One entry per location. What a resource touched inside it is listed by the kind-specific
subclass. Only facts that some locations genuinely lack are not required, so a row keeps the
same shape whatever it describes.
kind
Location type: dataset, sql_database, filesystem, rest_api, or a custom value.
resource_name
Resource that read from or wrote to this location, authoritative when metrics are collected.
location
Non-secret scope of the location, e.g. postgresql://example.com:5432, s3://bucket.
The key is absent when the location has no public address, for example a reverse ETL sink.
version
Version of the location's contents as a whole.
TDatasetDataLocation Objects
class TDatasetDataLocation(TDataLocation)
A dlt dataset: one or more schemas, holding tables, in a destination.
schemas
Schemas grouping the tables, each carrying its own version hash.
tables
Tables touched, as plain names - dlt qualifies tables by dataset, not by schema.
destination_fingerprint
Identifies destinations whose location is not public, ie. motherduck. May be empty.
casefold
Name of the casefolding function the destination applies: upper, lower or str.
case_sensitive
Whether the destination generates case sensitive identifiers, as adjusted at runtime.
dataset_name
Logical dataset name as configured by the user, e.g. My_DataSet. Absent for sinks.
physical_dataset_name
Normalized name as it exists in the store, e.g. my_data_set. Absent for sinks.
data_location_version
def data_location_version(schemas: Sequence[TSchemaReference]) -> str
Hashes the version hashes of schemas into a single version of the location contents.
ExtractMetrics Objects
class ExtractMetrics(StepMetrics)
job_metrics
Metrics collected per job id during writing of job file
table_metrics
Job metrics aggregated by table
resource_metrics
Job metrics aggregated by resource
dag
A resource dag where elements of the list are graph edges
hints
Hints passed to the resources
inputs
Locations read from, one entry per (resource, location)
NormalizeMetrics Objects
class NormalizeMetrics(StepMetrics)
job_metrics
Metrics collected per job id during writing of job file
table_metrics
Job metrics aggregated by table
LoadMetrics Objects
class LoadMetrics(StepMetrics)
dataset_name
Physical dataset name, normalized as it exists in the destination
outputs
Locations written to, one entry per (resource, location)