Skip to main content
Version: devel

dlt.common.storages.configuration

SchemaStorageConfiguration Objects

@configspec
class SchemaStorageConfiguration(BaseConfiguration)

View source on GitHub

schema_volume_path

path to volume with default schemas

import_schema_path

path from which to import a schema into storage

export_schema_path

path to which export schema from storage

external_schema_format

format in which to expect external schema

NormalizeStorageConfiguration Objects

@configspec
class NormalizeStorageConfiguration(BaseConfiguration)

View source on GitHub

normalize_volume_path

path to volume where normalized loader files will be stored

ensure_canonical_az_url

def ensure_canonical_az_url(bucket_url: str,
target_scheme: str,
storage_account_name: str = None,
account_host: str = None) -> str

View source on GitHub

Converts any of the forms of azure blob storage into canonical form of {target_scheme}://<container_name>@<storage_account_name>.{account_host}/path

azure_storage_account_name is optional only if not present in bucket_url, account_host assumes "<azure_storage_account_name>.dfs.core.windows.net" by default

make_fsspec_url

def make_fsspec_url(scheme: str, fs_path: str, bucket_url: str) -> str

View source on GitHub

Creates url from fs_path and scheme using bucket_url as an url template

Arguments:

  • scheme str - scheme of the resulting url
  • fs_path str - kind of absolute path that fsspec uses to locate resources for particular filesystem.
  • bucket_url str - an url template. the structure of url will be preserved if possible

FilesystemConfiguration Objects

@configspec
class FilesystemConfiguration(BaseConfiguration)

View source on GitHub

A configuration defining filesystem location and access credentials.

When configuration is resolved, bucket_url is used to extract a protocol and request corresponding credentials class.

  • s3
  • gs, gcs
  • az, abfs, adl, abfss, azure
  • file, memory
  • gdrive
  • sftp

read_only

Indicates read only filesystem access. Will enable caching

protocol

@property
def protocol() -> str

View source on GitHub

bucket_url protocol

pathlib

@property
def pathlib() -> Any

View source on GitHub

Returns pathlib suitable for joining and other path ops

normalize_bucket_url

def normalize_bucket_url() -> None

View source on GitHub

Normalizes bucket_url ie. by making local paths absolute and converting to file:

original_bucket_url

def original_bucket_url() -> str

View source on GitHub

Returns bucket_url before normalization

fingerprint

def fingerprint() -> str

View source on GitHub

Returns a fingerprint of bucket schema and netloc.

Returns:

  • str - Fingerprint.

make_url

def make_url(fs_path: str) -> str

View source on GitHub

Makes a full url (with scheme) form fs_path which is kind-of absolute path used by fsspec to identify resources. This method will use bucket_url to infer the original form of the url.

__str__

def __str__() -> str

View source on GitHub

Return displayable destination location

is_local_path

@staticmethod
def is_local_path(url: str) -> bool

View source on GitHub

Checks if url is a local path, without a schema

make_local_path

@staticmethod
def make_local_path(file_url: str) -> str

View source on GitHub

Gets a valid local filesystem path from file:// scheme. Supports POSIX/Windows/UNC paths

Returns:

  • str - local filesystem path

make_file_url

@staticmethod
def make_file_url(local_path: str) -> str

View source on GitHub

Creates a normalized file:// url from a local path

netloc is never set. UNC paths are represented as file://host/path

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.