Job configuration

This page documents the per-job options that aren't about when a job runs (those live in Triggers and scheduling) but about how it runs — execution limits, the Python environment it gets, and the configuration values it reads at runtime.

All options below are arguments to the @run.pipeline, @run.job, and @run.interactive decorators.

Execution constraints

execute={"timeout": "6h"} overrides the default 120-minute job timeout. Use the dict form to also customize the grace period — the window for the job to finish in-flight work before the dltHub platform hard-kills the process:

@run.pipeline(
    my_pipeline,
    execute={"timeout": 7200, "grace_period": 60},
)
def long_load():
    ...

Accepted timeout formats: a duration string ("6h", "30m") or an integer number of seconds.

Dependency groups

Install extra packages only for the jobs that need them. Declare a group in pyproject.toml:

[dependency-groups]
ibis = ["ibis-framework[duckdb]"]

Then opt into it in the decorator:

@run.pipeline(my_pipeline, require={"dependency_groups": ["ibis"]})
def transform(run_context: TJobRunContext):
    ...

The dltHub platform composes the execution environment from the workspace's base dependencies plus the job's declared groups.

Job configuration via TOML

Jobs read configuration through dlt's standard config system. The default section is the containing module name:

# applies to every job defined in usgs_pipeline.py
[jobs.usgs_pipeline]
epoch = "2026-04-05T00:00:00+00:00"

# overrides for one specific job
[jobs.usgs_pipeline.usgs_daily]
epoch = "2026-04-10T00:00:00+00:00"

For inline jobs in __deployment__.py, pass section="my_job" to the decorator to give it a clean section name. Profile-aware overrides live in dev.config.toml, prod.config.toml, etc. — see Workspace setup.

Display metadata

expose={...} controls how the job appears in the dashboard and to selectors:

@run.pipeline(
    "github_pipeline",
    expose={
        "tags": ["ingest"],
        "display_name": "GitHub commits ingest",
    },
)
def load_commits():
    ...

Key	Purpose
tags	List of labels for grouping in the dashboard and matching CLI selectors (tag:ingest)
display_name	Human-readable label shown in the dashboard

See Tags and bulk triggering for how tags drive dlthub job trigger selectors.

Next steps

• Triggers and scheduling — schedule and chain jobs
• Deployments — push these decorators to the dltHub platform