Iceberg-First Ingestion: How Taktile cut 70% of costs
Adrian Brudaru,
Co-Founder & CDO
Simon from Taktile shared a story that many data teams will recognize. His team's event data grew 12x in a single year, and their existing infrastructure couldn't keep up. What made his talk interesting wasn't the technology they chose, but how they thought about the problem and where they decided to make changes.
Watch the recording here, read the summary below.
The Problem: When growth breaks what used to work
Taktile's data architecture started simple and sensible. Events from their API flowed through Lambda functions into Snowflake using standard INSERT statements. This worked fine when they were processing modest volumes, but as their fintech platform grew, the numbers started telling a different story.
When Simon benchmarked their system, he found they could send 100 events per second to their pipeline, but Snowflake could only process about 30 of them. This meant that every night they would have a backlog of daily events that need to be cleared.
With 1,000 events per second on the horizon, something had to change. It wasn't just a matter of how to load to snowflake but also a concern on how the cost of loading 10x more data would look.
The Solution: Moving the heavy lifting
Rather than making any big or expensive changes to their loading or stack, Simon's team introduced Apache Iceberg tables as an intermediate layer between their Lambda functions and Snowflake. The insight was simple: let Lambda functions handle the raw data processing and storage, then let Snowflake focus on what it does best - analytics and queries.
Their approach was straightforward. Lambda functions now write events directly to Iceberg tables stored in S3, using dlt to handle the schema management and metadata. Snowflake accesses this data through catalog integration, so from an analytics perspective, nothing changed. But from a cost and performance perspective, everything changed.
Simon demonstrated this with a 50-line Lambda function that uses dlt to create Iceberg tables from incoming events. The key insight was separating concerns: Lambda functions scale horizontally to handle ingestion volume, while Snowflake compute is reserved for actual analysis work.
Taking it further: Local transformations
The team didn't stop at just moving the ingestion. They also rethought how they handle transformations. Instead of running all their dbt models in Snowflake, they started pulling raw data from the Iceberg tables, running transformations locally using DuckDB, and pushing only the final results back to Snowflake.
Simon showed this working live during his demo. He pulled raw event data from S3, transformed it on his laptop, and pushed clean dimension and fact tables to Snowflake. The intermediate processing happened in memory on commodity hardware instead of expensive warehouse compute.
This pattern proved particularly valuable for their data science workflows. Using dlt's Python SDK, data scientists could access the Iceberg-backed datasets as pandas DataFrames and build models without needing to understand the underlying infrastructure complexity.
The Results: 70% cost reduction
The numbers spoke for themselves. Taktile reduced their warehouse compute costs by 70% for data loading operations. They still pay for Lambda compute, but eliminated the expensive pattern where both Lambda functions and Snowflake were processing the same raw data.
More importantly, they eliminated the processing backlogs that had been building up throughout each day. Events now flow through the system at the same rate they arrive, and the team no longer worries about overnight catch-up processing.
For analytics and reporting, nothing changed from the user perspective. Snowflake still provides the same query performance and optimization, but now it's only doing the work that actually requires a data warehouse.
Why this approach worked
Simon identified three principles that made their implementation successful. The solution was modular - it integrated with their existing Python and dbt workflows without requiring a complete rebuild. It was built on open standards like Apache Iceberg and DuckDB, which gave them flexibility and avoided vendor lock-in. And it was portable - team members could run the entire pipeline locally for development and testing.
But the real insight was about timing and incremental change. They made the switch before their system broke completely, which allowed for a smoother transition. And they didn't try to solve everything at once - they kept their existing Snowflake investment while solving their specific scaling challenge.
What other teams can learn
For teams facing similar scaling challenges, Taktile's experience shows that you don't always need to rebuild everything from scratch. Sometimes the solution is recognizing which parts of your system are doing work they don't need to do, and finding ways to move that work to more appropriate places.
The key insight is that data warehouses are optimized for analytical workloads, not necessarily for high-volume ingestion. By moving the ingestion work to Lambda functions and Iceberg tables, Taktile let each component do what it does best, while dramatically reducing costs and improving performance.