Getting started

info

The Python SDK and the REST API are in public preview. Their interfaces will remain stable from this point onward and will only be extended. The packaging of the SDK may still change: it can move to another PyPI package and another top-level package name than dlthub_sdk.

The Python SDK is the dlthub_sdk package. It ships with dlthub-client, which dlt[hub] installs:

pip install "dlt[hub]"

Connect with an API key. The client closes when the with block ends:

import dlthub_sdk

with dlthub_sdk.connect(token="dlt_u_...") as runtime:
    for organization in runtime.organizations.list():
        for workspace in organization.workspaces.list():
            print(organization.name, workspace.id, workspace.name)

Every object leads to the objects that belong to it, so you go from the client to a workspace, from the workspace to a job, and from the job to its runs and their logs:

with dlthub_sdk.connect(token="dlt_u_...") as runtime:
    workspace = runtime.workspaces.get(id="<workspace id>")
    job = workspace.jobs.get_by_ref("jobs.load")

    run = job.start_run()
    for line in run.stream_logs():
        print(line.content)
    run = run.wait()
    print(run.status)

    for past_run in job.runs.list(limit=5):
        print(past_run.number, past_run.status, past_run.duration_seconds)

A workspace also reaches its variables, deployments, configurations and pipeline telemetry:

with dlthub_sdk.connect(token="dlt_u_...") as runtime:
    workspace = runtime.workspaces.get(id="<workspace id>")

    workspace.variables.set(
        "SOURCES__API__TOKEN", "...", profile=dlthub_sdk.WORKSPACE_PROFILE, secret=True
    )
    print(workspace.deployments.latest().version)
    for pipeline in workspace.telemetry.pipelines():
        print(pipeline.pipeline_name, pipeline.latest_status, pipeline.total_rows_loaded)

connect_async() gives the same objects for asyncio. Methods that call the platform return awaitables and lists are async iterables:

async with dlthub_sdk.connect_async(token="dlt_u_...") as runtime:
    workspace = await runtime.workspaces.get(id="<workspace id>")
    job = await workspace.jobs.get_by_ref("jobs.load")
    run = await job.start_run()
    async for line in run.stream_logs():
        print(line.content)

Collections and attributes

Each object in this reference lists its collections and its attributes:

• Collections lead to the objects that belong to it, such as workspace.jobs or job.runs. list() iterates over their objects and get() fetches one by its ID; most can also count() them. A few, such as workspace.variables and workspace.telemetry, group operations instead of holding objects.
• Attributes are the data the object carries. to_dict() returns them as a plain dict.

Types in this reference are those of the blocking client. API keys, base URLs and the data plane are explained in the REST API reference, which the SDK is built on.

This reference is generated from dlthub-client 0.28.8a3.