dlt.destinations.impl.duckdb.configuration

DuckDbBaseCredentials Objects

@configspec(init=False)
class DuckDbBaseCredentials(CredentialsConfiguration)

View source on GitHub

read_only

Open database r or rw

extensions

Extensions loaded on each newly opened connection

global_config

Global config applied once on each newly opened connection

pragmas

Pragmas set applied to each borrowed connection

local_config

Local config applied to each borrowed connection

DuckDbConnectionPool Objects

class DuckDbConnectionPool()

View source on GitHub

always_open_connection

Always opens a new connection without cloning with cursor

__init__

def __init__(credentials: DuckDbBaseCredentials,
             always_open_connection: bool = False)

View source on GitHub

Initializes a connection pool that dispenses duckdb connection to be used in multiple threads.

Default mode of the operation is to create a single duckdb connection and then use duplicate method to pass a connection clone to a thread.

With always_open_connection, thread receives a new duckdb connection every time primarily to support attached databases like ducklake. Current implementation does not pool connections in this mode, it creates a fresh copy on each request.

This mechanism is piggybacking on destination Configuration/Credentials which are a singleton in pipeline in load step. This allows to dispense connections in to workers in multiple threads.

borrow_conn

def borrow_conn(global_config: Dict[str, Any] = None,
                local_config: Dict[str, Any] = None,
                pragmas: List[str] = None) -> DuckDBPyConnection

View source on GitHub

Opens new or clones existing duckdb connection to support multi-thread access and then borrows it to the caller. Caller is supposed to return the connection when it is no longer needed. If connection is not returned the underlying duckdb conn will never be closed due to internal ref counting.

return_conn

def return_conn(borrowed_conn: DuckDBPyConnection) -> int

View source on GitHub

Closed the borrowed conn, if refcount goes to 0, duckdb connection is deleted

move_conn

def move_conn() -> DuckDBPyConnection

View source on GitHub

Takes ownership of the connection so it won't be closed on refcount 0 and in destructor

never_borrowed

@property
def never_borrowed() -> bool

View source on GitHub

Returns true if connection was not yet created or no connections were borrowed in case of external connection

DuckDbCredentials Objects

@configspec
class DuckDbCredentials(DuckDbBaseCredentials, ConnectionStringCredentials)

View source on GitHub

drivername

type: ignore

__init__

def __init__(conn_or_path: Union[str, DuckDBPyConnection] = None,
             *,
             read_only: bool = False,
             extensions: Optional[List[str]] = None,
             global_config: Optional[Dict[str, Any]] = None,
             pragmas: Optional[List[str]] = None,
             local_config: Optional[Dict[str, Any]] = None) -> None

View source on GitHub

Initialize DuckDB credentials with a connection or file path and connection settings.

Arguments:

• conn_or_path - Either a DuckDB connection object or a path to a DuckDB database file. Can also be special values like ':pipeline:' or ':memory:'.
• read_only - Open database in read-only mode if True, read-write mode if False
• extensions - List of DuckDB extensions to load on each newly opened connection
• global_config - Dictionary of global configuration settings applied once on each newly opened connection
• pragmas - List of PRAGMA statements to be applied to each cursor connection
• local_config - Dictionary of local configuration settings applied to each cursor connection

DuckDbClientConfiguration Objects

@configspec
class DuckDbClientConfiguration(WithLocalFiles,
                                DestinationClientDwhWithStagingConfiguration)

View source on GitHub

destination_type

type: ignore