Skip to main content
Version: devel

dlt.extract.reference

SourceFactory Objects

class SourceFactory(ABC, Generic[TSourceFunParams, TDltSourceImpl])

View source on GitHub

__call__

def __call__(*args: TSourceFunParams.args,
**kwargs: TSourceFunParams.kwargs) -> TDltSourceImpl

View source on GitHub

Makes dlt source

clone

@abstractmethod
def clone(*,
name: str = None,
section: str = None,
max_table_nesting: int = None,
root_key: bool = False,
schema: Schema = None,
schema_contract: TSchemaContract = None,
spec: Type[BaseConfiguration] = None,
parallelized: bool = None,
_impl_cls: Type[TDltSourceImpl] = DltSource) -> Self

View source on GitHub

Overrides default decorator arguments that will be used to when DltSource instance and returns modified clone.

SourceReference Objects

class SourceReference()

View source on GitHub

SOURCES

A registry of all the decorated sources and resources discovered when importing modules

ref

A fully qualified reference: module.name to the factory instance

expand_shorthand_ref

@classmethod
def expand_shorthand_ref(cls, ref: str) -> List[str]

View source on GitHub

Converts ref into fully qualified form, return one or more alternatives for shorthand notations. Run context is injected if needed. Following formats are recognized

  • section.name
  • name

find

@classmethod
def find(cls,
ref: str,
raise_exec_errors: bool = False,
import_missing_modules: bool = False,
_impl_sig: Callable[TSourceFunParams, Any] = None,
_impl_cls: Type[TDltSourceImpl] = None) -> Any

View source on GitHub

Returns source factory from reference ref. Looks into registry or tries auto-import. Expands shorthand notation into section.name eg. "sql_database" is expanded into "dlt.sources.sql_database.sql_database".

You can control auto-import behavior:

  • raise_exec_errors - will re-raise code execution errors in imported modules
  • import_missing_modules - will ignore missing dependencies during import by substituting them with dummy modules. this should be only used to manipulate local dev environment

from_reference

@classmethod
def from_reference(cls,
ref: str,
name: str = None,
section: str = None,
max_table_nesting: int = None,
root_key: bool = False,
schema: Schema = None,
schema_contract: TSchemaContract = None,
spec: Type[BaseConfiguration] = None,
parallelized: bool = None,
_impl_cls: Type[TDltSourceImpl] = None,
source_args: Tuple[Any, ...] = None,
source_kwargs: Dict[str, Any] = None) -> TDltSourceImpl

View source on GitHub

Find registered source factory or imports it, then instantiates the DltSource using passed args and kwargs. Passes additional arguments to clone of source factory

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.