Highlights
- Democratizing data access: With Snowflake and dlt, an open-source Python library for data loading, Taktile can reduce bottlenecks. If any Taktile team, whether engineering or business, wants to innovate, they don’t need to wait for permission or data access. Now Taktile’s teams can easily create and process datasets in Python.
- Enhanced compliance and security: Taktile securely manages data ingestion within their in-house infrastructure, powered by dlt + Snowflake: dlt is running wherever Python runs and Snowflake is leveraged as a secure warehouse for data storage. This approach ensures compliance with regulatory standards such as GDPR, CCPA, ISO 27001, SOC 2, data localization laws, and other security protocols, and safeguards sensitive information.
- Reducing costs: Taktile significantly reduced costs by completely transforming its data management processes, harnessing the capabilities of dlt + Snowflake. They not only cut costs from a cloud ETL provider, but optimized expenses for data and code maintenance.
Data Stack
Data sources: custom sources, including Hubspot, Github, Linear, Betteruptime, event streaming
Destination: Snowflake
Orchestration: Github Actions, AWS Lambda
Transformation: dbt
Python and Machine Learning under security constraints are key to our success. We found that our cloud ETL provider could not meet our needs. For us, Snowflake is a secure warehouse for data storage. dlt is a lightweight yet powerful open source tool we can run together with Snowflake. Our event streaming and batch data loading performs at scale and low cost. Now anyone who knows Python can self-serve to fulfill their data needs.
- Max Eber, CPTO, Taktile
Challenge
The company manages and analyses vast amounts of sensitive data of its customers.
Keeping customer data safe and secure is a key priority for Taktile. Therefore, Taktile only loads a subset of non-sensitive events into its warehouse and cannot rely on external vendors to access decision data.
Before using Snowflake + dlt, the company used an expensive cloud ETL provider with limited connectors that were hard to customize to their needs. Customizing and updating existing data sources took up to 6 months.
As with many financial technology companies, the use of Python and Machine Learning is key to the company’s success. Taktile was running home-grown Python scripts to meet its needs that were not served by the cloud ETL provider. Setting up external stages in Snowflake for these scripts was very laborious, time-consuming, and error-prone at this scale.
The company’s small data team was stretched and became a bottleneck. Any time anyone in the company needed access to particular data, they had to go through the data team.
Solution
Trying out dlt + snowflake for custom data needs and empowering all software engineers
The initial motivation to try dlt happened when Taktile’s CPTO, Maximilian Eber, decided to further invest in customer experience and incident management. This effort required the collection of more data on how Taktile is being used by its customers, but such a connector was not available in Taktile’s cloud ETL vendor.
Instead of creating yet another hard-to-maintain Python script, Taktile’s Lead Data Engineer Simon Bumm created dlt custom sources for Linear, Github, Betteruptime and AWS Cost Explorer.
Expanding to a lightweight data platform
After that, Taktile went on to start building a lightweight data platform.
To process millions of daily tracking events, the company turned to dlt, AWS Lambda and Snowflake. With dlt and AWS Lambda serving as the backbone of data ingestion, anonymization, normalization, and loading processes, Simon achieved a seamless integration with Snowflake. Real-time event streaming and batch data loading mechanisms were orchestrated through dlt, ensuring secure and compliant data ingestion. This in-house infrastructure not only eliminated dependencies on external vendors but also safeguarded sensitive financial data.
As a further step, the company decided to address customization limits of the Hubspot connector from the cloud ETL provider it was using. The off-the-shelf connector could not be customized to the level of Taktile’s needs, causing daily issues that Solutions Engineering had to tackle. In contrast, the company was able to customize a dlt Hubspot source to its needs quickly.
It saves us a lot of time not having to create external stages and tables for each new data source that we connect to Snowflake. We set up the destination and staging once in dlt and it “just works” for each pipeline. All of our software engineers are able to see any incidents, quickly analyze them in a deeper way and respond to them quicker. I stopped being the gatekeeper.
- Simon Bumm, Lead Data Engineer, Taktile
Results
The implementation of dlt and Snowflake yielded remarkable results for Taktile:
1. Efficient data ingestion: Taktile successfully loads millions of events and user actions into Snowflake daily, ensuring efficient data management. The platform's architecture enables effortless scalability to accommodate irregular traffic volumes, guaranteeing timely event ingestion even under high loads.
2. Compliance and security: One of the significant outcomes of Taktile's adoption of the new lightweight data platform is enhanced compliance and security measures. By leveraging dlt and Snowflake, Taktile ensures that the data ingestion and analysis processes adhere to strict regulatory standards and security protocols. Because dlt operates wherever Python runs, Taktile could host all data ingestion infrastructure internally with no exposure of sensitive information to third-party entities.
3. Elimination of bottlenecks: The data team transitioned from being a bottleneck to facilitating seamless data access for all teams within the organization, fostering collaboration and efficiency. The number of engineers capable of integrating new data into Snowflake expanded from skilled data engineers to nearly anyone within the company. For example, the customer support team recently added a connector to Thena, their support platform, within a day.
4. Cost optimization: Taktile achieved a significant reduction in costs, lowering expenses by 20x while enhancing data processing capabilities.
5. Flexibility and collaboration: The platform enables rapid customization and integration of custom data sources within hours, empowering ML engineers and data analytics to derive actionable insights swiftly.
Future
Looking ahead, Taktile has already planned to leverage Snowflake's comprehensive capabilities by migrating data ingestion processes to the Snowflake container service. This strategic move is aimed at optimizing resource utilization and enhancing scalability.
Furthermore, as part of Taktile's ongoing efforts towards data democratization, the third crucial phase involves providing Taktile’s customers with a flexible and scalable way of exporting their historical Taktile decision data to their data platform of choice.
About Taktile
Taktile is the leading decision management platform for the financial services industry. Taktile helps its customers with credit underwriting, fraud checks, transaction and account monitoring, onboarding & and KYC/KYB (Know Your Customer/Business). Credit and risk teams across the globe use Taktile’s low-code solution to design, build, and evaluate automated decision flows at scale.