dlt.destinations.impl.duckdb.configuration
NON_ATTACHABLE_LOCATIONS
Databases that live inside a single connection. No other connection can attach them.
ConnStatement Objects
class ConnStatement(NamedTuple)
A database-scoped statement a pool runs on each connection it opens
key
What the statement configures. The default is the SQL itself. The pool replaces all statements with the same key as one group
DuckDbBaseCredentials Objects
@configspec(init=False)
class DuckDbBaseCredentials(CredentialsConfiguration)
read_only
Open database r or rw
extensions
Extensions loaded on each newly opened connection
global_config
Global config applied once on each newly opened connection
pragmas
Pragmas set applied to each borrowed connection
statements
Database-scoped SQL run on each newly opened connection, after extensions and
global_config. This field holds the statements those two fields cannot express, such as
INSTALL, ATTACH and CREATE SECRET
local_config
Local config applied to each borrowed connection
session_timezone
TimeZone set on each newly opened connection, which its sessions inherit. None keeps
the duckdb default
external_conn
def external_conn() -> Optional[DuckDBPyConnection]
Returns the connection that the caller passed, None when dlt opens its own.
DuckDbConnectionPool Objects
class DuckDbConnectionPool()
always_open_connection
Always opens a new connection without cloning with cursor
__init__
def __init__(credentials: DuckDbBaseCredentials,
always_open_connection: bool = False)
Initializes a connection pool that dispenses duckdb connection to be used in multiple threads.
Default mode of the operation is to create a single duckdb connection and then use duplicate
method to pass a connection clone to a thread.
With always_open_connection, thread receives a new duckdb connection every time primarily
to support attached databases like ducklake. Current implementation does not pool connections
in this mode, it creates a fresh copy on each request.
This mechanism is piggybacking on destination Configuration/Credentials which are a singleton in pipeline in load step. This allows to dispense connections in to workers in multiple threads.
borrow_conn
def borrow_conn(global_config: Dict[str, Any] = None,
local_config: Dict[str, Any] = None,
pragmas: List[str] = None) -> DuckDBPyConnection
Opens new or clones existing duckdb connection to support multi-thread access and then borrows it to the caller. Caller is supposed to return the connection when it is no longer needed. If connection is not returned the underlying duckdb conn will never be closed due to internal ref counting.
add_statements
def add_statements(statements: Sequence[ConnStatement],
alias: str = None,
conn: DuckDBPyConnection = None) -> List[ConnStatement]
Registers statements to run on each connection this pool opens. Returns the
statements that are new.
Arguments:
statements- Database-scoped statements, seeConnStatement.alias- The attach catalog that the statements add. The pool keeps it inattached_aliases.conn- The connection that gets the statements at once. This argument is necessary only withalways_open_connection, where the pool keeps no connection of its own.
return_conn
def return_conn(borrowed_conn: DuckDBPyConnection) -> int
Closed the borrowed conn, if refcount goes to 0, duckdb connection is deleted
move_conn
def move_conn() -> DuckDBPyConnection
Takes ownership of the connection so it won't be closed on refcount 0 and in destructor
DuckDbCredentials Objects
@configspec
class DuckDbCredentials(DuckDbBaseCredentials, ConnectionStringCredentials)
drivername
type: ignore
__init__
def __init__(conn_or_path: Union[str, DuckDBPyConnection] = None,
*,
read_only: bool = False,
extensions: Optional[List[str]] = None,
global_config: Optional[Dict[str, Any]] = None,
pragmas: Optional[List[str]] = None,
statements: Optional[List[str]] = None,
local_config: Optional[Dict[str, Any]] = None,
session_timezone: Optional[str] = "UTC") -> None
Initialize DuckDB credentials with a connection or file path and connection settings.
Arguments:
conn_or_path- Either a DuckDB connection object or a path to a DuckDB database file. Can also be special values like ':pipeline:' or ':memory:'.read_only- Open database in read-only mode if True, read-write mode if Falseextensions- List of DuckDB extensions to load on each newly opened connectionglobal_config- Dictionary of global configuration settings applied once on each newly opened connectionpragmas- List of PRAGMA statements to be applied to each cursor connectionstatements- Database-scoped SQL run on each newly opened connection, for exampleINSTALL,ATTACHandCREATE SECRET. Session settings belong inpragmasorlocal_configlocal_config- Dictionary of local configuration settings applied to each cursor connectionsession_timezone-TimeZoneset on each newly opened connection, which its cursor connections inherit.Nonekeeps the duckdb default
DuckDbClientConfiguration Objects
@configspec
class DuckDbClientConfiguration(WithAttachableEngine, WithLocalFiles,
DestinationClientDwhWithStagingConfiguration)
destination_type
type: ignore
data_location
def data_location() -> str
Returns the database file path. For a database that lives inside a query engine, returns the marker of that database and the identity of the engine.
needs_attach
def needs_attach(other: DestinationClientConfiguration) -> bool
Returns False for a database that this query engine already opened. The engine accesses every schema of that database.
attach_type
def attach_type() -> Optional[TAttachType]
Returns None for a database that lives inside a query engine. Such a database has no path to attach.