Python dlt
Library: Loading Data from slack
to snowflake
Join our Slack community or book a call with our support engineer Violetta.
This is a technical guide for utilizing dlt
, an open-source Python library, to load data from Slack
to Snowflake
. Slack
is a business messaging app that centralizes information flow, while Snowflake
is a cloud-based data warehousing platform that excels in storing, processing, and analyzing large data volumes. By leveraging dlt
, we can efficiently transport data from Slack
to Snowflake
for further analysis. For additional information about Slack
, please visit https://slack.com.
dlt
Key Features
Scalability and Performance:
dlt
offers several mechanisms and configuration options to scale up and fine-tune pipelines, such as running extraction, normalization, and load in parallel, and writing sources and resources that run in parallel via thread pools and async execution. Read more about performance.Data Governance:
dlt
pipelines provide robust governance support through utilization of pipeline metadata, schema enforcement and curation, and schema change alerts, contributing to better data management practices, compliance adherence, and overall data governance. Read more about governance.Data Extraction: Extracting data with
dlt
is simple and scalable, leveraging iterators, chunking, and parallelization techniques, and utilizing implicit extraction DAGs for efficient API calls for data enrichments or transformations. Read more about data extraction.Snowflake Destination:
dlt
supports Snowflake as a destination, accepting three authentication types and providing detailed instructions for each. Read more about Snowflake destination.Community Support:
dlt
has a growing community that supports many features and use cases needed by the community. You can join their Slack community to find recent releases or discuss what you can build withdlt
. Join thedlt
community.
Getting started with your pipeline locally
0. Prerequisites
dlt
requires Python 3.8 or higher. Additionally, you need to have the pip
package manager installed, and we recommend using a virtual environment to manage your dependencies. You can learn more about preparing your computer for dlt in our installation reference.
1. Install dlt
First you need to install the dlt
library with the correct extras for Snowflake
:
pip install "dlt[snowflake]"
The dlt
cli has a useful command to get you started with any combination of source and destination. For this example, we want to load data from Slack
to Snowflake
. You can run the following commands to create a starting point for loading data from Slack
to Snowflake
:
# create a new directory
mkdir slack_pipeline
cd slack_pipeline
# initialize a new pipeline with your source and destination
dlt init slack snowflake
# install the required dependencies
pip install -r requirements.txt
The last command will install the required dependencies for your pipeline. The dependencies are listed in the requirements.txt
:
dlt[snowflake]>=0.3.12
You now have the following folder structure in your project:
slack_pipeline/
├── .dlt/
│ ├── config.toml # configs for your pipeline
│ └── secrets.toml # secrets for your pipeline
├── slack/ # folder with source specific files
│ └── ...
├── slack_pipeline.py # your main pipeline script
├── requirements.txt # dependencies for your pipeline
└── .gitignore # ignore files for git (not required)
2. Configuring your source and destination credentials
The dlt
cli will have created a .dlt
directory in your project folder. This directory contains a config.toml
file and a secrets.toml
file that you can use to configure your pipeline. The automatically created version of these files look like this:
generated config.toml
# put your configuration values here
[runtime]
log_level="WARNING" # the system log level of dlt
# use the dlthub_telemetry setting to enable/disable anonymous usage data reporting, see https://dlthub.com/docs/telemetry
dlthub_telemetry = true
generated secrets.toml
# put your secret values and credentials here. do not share this file and do not push it to github
[sources.slack]
access_token = "access_token" # please set me up!
[destination.snowflake.credentials]
database = "database" # please set me up!
password = "password" # please set me up!
username = "username" # please set me up!
host = "host" # please set me up!
warehouse = "warehouse" # please set me up!
role = "role" # please set me up!
2.1. Adjust the generated code to your usecase
3. Running your pipeline for the first time
The dlt
cli has also created a main pipeline script for you at slack_pipeline.py
, as well as a folder slack
that contains additional python files for your source. These files are your local copies which you can modify to fit your needs. In some cases you may find that you only need to do small changes to your pipelines or add some configurations, in other cases these files can serve as a working starting point for your code, but will need to be adjusted to do what you need them to do.
The main pipeline script will look something like this:
"""Pipeline to load slack into duckdb."""
from typing import List
import dlt
from pendulum import datetime
from slack import slack_source
def load_all_resources() -> None:
"""Load all resources from slack without any selection of channels."""
pipeline = dlt.pipeline(
pipeline_name="slack", destination='snowflake', dataset_name="slack_data"
)
source = slack_source(
page_size=1000, start_date=datetime(2023, 9, 1), end_date=datetime(2023, 9, 8)
)
# Uncomment the following line to load only the access_logs resource. It is not selectes
# by default because it is a resource just available on paid accounts.
# source.access_logs.selected = True
load_info = pipeline.run(
source,
)
print(load_info)
def select_resource(selected_channels: List[str]) -> None:
"""Execute a pipeline that will load the given Slack list of channels with the selected
channels incrementally beginning at the given start date."""
pipeline = dlt.pipeline(
pipeline_name="slack", destination='snowflake', dataset_name="slack_data"
)
source = slack_source(
page_size=20,
selected_channels=selected_channels,
start_date=datetime(2023, 9, 1),
end_date=datetime(2023, 9, 8),
).with_resources("channels", "1-announcements", "dlt-github-ci")
load_info = pipeline.run(
source,
)
print(load_info)
def get_users() -> None:
"""Execute a pipeline that will load Slack users list."""
pipeline = dlt.pipeline(
pipeline_name="slack", destination='snowflake', dataset_name="slack_data"
)
source = slack_source(
page_size=20,
).with_resources("users")
load_info = pipeline.run(
source,
)
print(load_info)
if __name__ == "__main__":
# Add your desired resources to the list...
# resources = ["access_logs", "conversations", "conversations_history"]
# load_all_resources()
# select_resource(selected_channels=["dlt-github-ci"])
# select_resource(selected_channels=["1-announcements", "dlt-github-ci"])
get_users()
Provided you have set up your credentials, you can run your pipeline like a regular python script with the following command:
python slack_pipeline.py
4. Inspecting your load result
You can now inspect the state of your pipeline with the dlt
cli:
dlt pipeline slack info
You can also use streamlit to inspect the contents of your Snowflake
destination for this:
# install streamlit
pip install streamlit
# run the streamlit app for your pipeline with the dlt cli:
dlt pipeline slack show
5. Next steps to get your pipeline running in production
One of the beauties of dlt
is, that we are just a plain Python library, so you can run your pipeline in any environment that supports Python >= 3.8. We have a couple of helpers and guides in our docs to get you there:
The Deploy section will show you how to deploy your pipeline to
- Deploy with Github Actions:
dlt
allows you to prepare your pipeline for deployment using GitHub Actions, a CI/CD runner that is basically free. You can specify when the GitHub Action should run using a cron schedule expression. Learn more about it here. - Deploy with Airflow:
dlt
can also be deployed with Airflow, a platform to programmatically author, schedule and monitor workflows. You can create an Airflow DAG for your pipeline script that you should customize. Detailed instructions can be found here. - Deploy with Google Cloud Functions:
dlt
also supports deployment with Google Cloud Functions, a serverless execution environment for building and connecting cloud services. You can find more about it here. - Other Deployment Options: Apart from the above mentioned options,
dlt
supports various other deployment options. You can find more about them here.
The running in production section will teach you about:
- Monitor Your Pipeline:
dlt
allows you to keep a close eye on your pipeline's performance and status. Get detailed insights about your pipeline's operation with the help of this guide. - Set Up Alerts: Stay informed about your pipeline's health by setting up alerts.
dlt
makes it easy to set up alerts to notify you of any issues in your pipeline. Learn how to set up alerts with this guide. - Set Up Tracing: Trace the execution of your pipeline to understand its behavior better.
dlt
provides powerful tracing capabilities to help you debug and optimize your pipeline. Find out how to set up tracing with this guide.
Additional pipeline guides
- Load data from Rest API to Snowflake in python with dlt
- Load data from Airtable to CockroachDB in python with dlt
- Load data from Salesforce to EDB BigAnimal in python with dlt
- Load data from AWS S3 to Google Cloud Storage in python with dlt
- Load data from Azure Cloud Storage to Timescale in python with dlt
- Load data from PostgreSQL to The Local Filesystem in python with dlt
- Load data from AWS S3 to ClickHouse in python with dlt
- Load data from Chess.com to DuckDB in python with dlt
- Load data from Google Cloud Storage to PostgreSQL in python with dlt
- Load data from Klarna to Azure Cloud Storage in python with dlt