Skip to main content
Version: 1.0.0 (latest)

sources.sql_database_pipeline

load_select_tables_from_database

def load_select_tables_from_database() -> None

[view_source]

Use the sql_database source to reflect an entire database schema and load select tables from it.

This example sources data from the public Rfam MySQL database.

load_entire_database

def load_entire_database() -> None

[view_source]

Use the sql_database source to completely load all tables in a database

load_standalone_table_resource

def load_standalone_table_resource() -> None

[view_source]

Load a few known tables with the standalone sql_table resource, request full schema and deferred table reflection

select_columns

def select_columns() -> None

[view_source]

Uses table adapter callback to modify list of columns to be selected

select_with_end_value_and_row_order

def select_with_end_value_and_row_order() -> None

[view_source]

Gets data from a table withing a specified range and sorts rows descending

my_sql_via_pyarrow

def my_sql_via_pyarrow() -> None

[view_source]

Uses pyarrow backend to load tables from mysql

create_unsw_flow

def create_unsw_flow() -> None

[view_source]

Uploads UNSW_Flow dataset to postgres via csv stream skipping dlt normalizer. You need to download the dataset from https://github.com/rdpahalavan/nids-datasets

test_connectorx_speed

def test_connectorx_speed() -> None

[view_source]

Uses unsw_flow dataset (~2mln rows, 25+ columns) to test connectorx speed

use_type_adapter

def use_type_adapter() -> None

[view_source]

Example use of type adapter to coerce unknown data types

specify_columns_to_load

def specify_columns_to_load() -> None

[view_source]

Run the SQL database source with a subset of table columns loaded

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.