dlt.extract.reference

SourceFactory Objects

class SourceFactory(ABC, Generic[TSourceFunParams, TDltSourceImpl])

View source on GitHub

name

Name of the source which is also the name of the associated schema.

section

Configuration section right after 'sources' in default layout.

max_table_nesting

Maximum depth of nested table above which remaining nodes are loaded as structs or JSON.

root_key

Enables merging on all resources by propagating root foreign key to nested tables.

schema

An explicit Schema instance to be associated with the source.

schema_contract

Schema contract settings that will be applied to this source.

spec

A specification of configuration and secret values required by the source.

parallelized

If True, resource generators will be extracted in parallel with other resources.

add_postprocessor

@abstractmethod
def add_postprocessor(
    func: Callable[[TDltSourceImpl], Union[TDltSourceImpl,
                                           Awaitable[TDltSourceImpl]]]
) -> None

View source on GitHub

Adds a callback that receives and returns a DltSource after it is created.

clone

@abstractmethod
def clone(*,
          name: str = None,
          section: str = None,
          max_table_nesting: int = None,
          root_key: bool = None,
          schema: Schema = None,
          schema_contract: TSchemaContract = None,
          spec: Type[BaseConfiguration] = None,
          parallelized: bool = None,
          _impl_cls: Type[TDltSourceImpl] = DltSource) -> Self

View source on GitHub

Overrides default decorator arguments that will be used to when DltSource instance and returns modified clone.

SourceReference Objects

class SourceReference()

View source on GitHub

SOURCES

A registry of all the decorated sources and resources discovered when importing modules

ref

A fully qualified reference: module.name to the factory instance

expand_shorthand_ref

@classmethod
def expand_shorthand_ref(cls, ref: str) -> List[str]

View source on GitHub

Converts ref into fully qualified form, return one or more alternatives for shorthand notations. Run context is injected if needed. Following formats are recognized

• section.name
• name

find

@classmethod
def find(cls,
         ref: str,
         raise_exec_errors: bool = False,
         import_missing_modules: bool = False,
         _impl_sig: Callable[TSourceFunParams, Any] = None,
         _impl_cls: Type[TDltSourceImpl] = None) -> Any

View source on GitHub

Returns source factory from reference ref. Looks into registry or tries auto-import. Expands shorthand notation into section.name eg. "sql_database" is expanded into "dlt.sources.sql_database.sql_database".

You can control auto-import behavior:

• raise_exec_errors - will re-raise code execution errors in imported modules
• import_missing_modules - will ignore missing dependencies during import by substituting them with dummy modules. this should be only used to manipulate local dev environment

from_reference

@classmethod
def from_reference(cls,
                   ref: str,
                   name: str = None,
                   section: str = None,
                   max_table_nesting: int = None,
                   root_key: bool = None,
                   schema: Schema = None,
                   schema_contract: TSchemaContract = None,
                   spec: Type[BaseConfiguration] = None,
                   parallelized: bool = None,
                   _impl_cls: Type[TDltSourceImpl] = None,
                   source_args: Tuple[Any, ...] = None,
                   source_kwargs: Dict[str, Any] = None) -> TDltSourceImpl

View source on GitHub

Find registered source factory or imports it, then instantiates the DltSource using passed args and kwargs. Passes additional arguments to clone of source factory