Skip to main content
Version: devel

dlt.extract.pipe

ForkPipe Objects

class ForkPipe(ItemTransform[ResolvablePipeItem])

View source on GitHub

__init__

def __init__(pipe: "Pipe", step: int = -1, copy_on_fork: bool = False) -> None

View source on GitHub

A transformer that forks the pipe and sends the data items to forks added via add_pipe method.

Pipe Objects

class Pipe(SupportsPipe)

View source on GitHub

is_empty

@property
def is_empty() -> bool

View source on GitHub

Checks if pipe contains any steps

is_data_bound

@property
def is_data_bound() -> bool

View source on GitHub

Checks if pipe is bound to data and can be iterated. Pipe is bound if has a parent that is bound xor is not empty.

gen

@property
def gen() -> TPipeStep

View source on GitHub

A data generating step

find

def find(*step_type: AnyType) -> int

View source on GitHub

Finds a step with object of type step_type

find_all

def find_all(*step_type: AnyType) -> List[int]

View source on GitHub

Finds all steps with object of type step_type

get_by_type

def get_by_type(*step_type: AnyType) -> TPipeStep

View source on GitHub

Gets first step found with object of type step_type

remove_by_type

def remove_by_type(*step_type: AnyType) -> int

View source on GitHub

Deletes first step found with object of type step_type, returns previous index

append_step

def append_step(step: TPipeStep) -> "Pipe"

View source on GitHub

Appends pipeline step. On first added step performs additional verification if step is a valid data generator

insert_step

def insert_step(step: TPipeStep, index: int) -> "Pipe"

View source on GitHub

Inserts step at a given index in the pipeline. Allows prepending only for transformers

remove_step

def remove_step(index: int) -> None

View source on GitHub

Removes steps at a given index. Gen step cannot be removed

replace_gen

def replace_gen(gen: TPipeStep) -> None

View source on GitHub

Replaces data generating step. Assumes that you know what are you doing

close

def close() -> None

View source on GitHub

Closes pipe generator

full_pipe

def full_pipe() -> "Pipe"

View source on GitHub

Creates a pipe that from the current and all the parent pipes.

ensure_gen_bound

def ensure_gen_bound() -> None

View source on GitHub

Verifies that gen step is bound to data

evaluate_gen

def evaluate_gen() -> None

View source on GitHub

Lazily evaluate gen of the pipe when creating PipeIterator. Allows creating multiple use pipes from generator functions and lists

bind_gen

def bind_gen(*args: Any, **kwargs: Any) -> Any

View source on GitHub

Finds and wraps with args + kwargs the callable generating step in the resource pipe and then replaces the pipe gen with the wrapped one

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.