The agentic data layer

As a simple-to-use Python library, dlt is the first tool that this new wave of people can use. By leveraging this library, we can extend the machine learning revolution into enterprise data.

Python and machine learning under security constraints are key to our success. dlt is a lightweight yet powerful open source tool we can run together with Snowflake. Now anyone who knows Python can self-serve to fulfil their data needs.

I am building our internal skills usage leaderboard so we can see how people are using AI and spread what's working. dltHub turns this into a cohesive process without messy scripts to dedupe queries or wrangle intermediate tables. And the best part is anyone on the team can use agents to easily contribute.

It's easy to write AI-assisted code and get a prototype. dltHub instead is built for the inspect-validate-debug loop, which is where AI-assisted data engineering actually lives or dies. The first platform I've seen that treats that loop as a design assumption, not a feature.

The modeling layer is where the real cost lives, and it's where consulting companies like ours have been doing the most repetitive, high-cost manual work. We are excited about the dltHub product vision for the modeling layer because the main challenge in all data projects is how the data is being interpreted.

In dltHub, agents propose transformations in Python, dlt.hub.transformation compiles them to SQL through Ibis, and execution stays inside DuckDB. The data doesn't move into Python memory at any step — the same composition pattern Ibis and Arrow are designed around.

Every Ops leader is trying to answer the same question: is my team getting better or worse? I always had good intuition about what data I needed, but never the resources to get it and measure it reliably. dltHub changes the equation: it's the first product I've seen built for operators and their agents, not just fully-staffed enterprise data teams.

dltHub and Snowflake deliver a simple, end-to-end pathway for financial institutions to transform raw data into governed analytics and AI-ready datasets without needing a full engineering team. You can pull data from core banking systems, market feeds, and APIs directly into Snowflake.

dltHub lets my agents develop data pipelines locally, test changes quickly and cheaply in CI, and then runs them in the cloud against my largest workloads. It gives them the tools to take care of the knucklehead stuff so that I can get a good night's sleep.

We weren't looking for a custom rebuild, and we didn't want to staff up a data team to run something we couldn't sustain at our size. The toolkit by dltHub took what we already had, structured the parts that needed cleaning up, and gave us back a stack we can confidently run, with Chat-BI on top that actually understands our business.

We needed analytics on our GitHub Actions CI. With dltHub we wired the end-to-end pipeline up in hours, not days. Now we see where builds drag, fix the slow spots, and ship faster.

The quality of AI output depends on the frameworks and context used. This is where the dltHub AI Workbench shines. Ingestion used to sit with me and one other person. Now our analytics engineers self-serve.

Ingestion moved from being owned by a small group with deep tool knowledge to something any Python developer on the team could author, review, and ship. The question changed from "who knows the tool?" to "what data do we need next?"

I just wanted to reiterate how damned cool dlthub and the agentic workflows continue to be. We have a mountain of Klaviyo data, and API rate limits made extraction slow and prone to failure. The agent devised a tiered chunking strategy: 7 days, dropping to 1 day, then 1 hour on timeout, and it's gotten through some highly problematic data patterns without missing a beat. Super impressive, you guys are amazing!