Loading Data from Cisco Meraki to CockroachDB Using dlt in Python
We will be using the dlt PostgreSQL destination to connect to CockroachDB. You can get the connection string for your CockroachDB database as described in the CockroachDB Docs.
Join our Slack community or book a call with our support engineer Violetta.
Cisco Meraki is a cloud-managed IT solution that simplifies the management of wireless, switching, security, and mobile device management. It provides powerful tools for network visibility, control, and automation. With an intuitive dashboard, Cisco Meraki allows organizations to manage their entire network from a single interface, enhancing operational efficiency and security. It offers features like real-time monitoring, remote troubleshooting, and automated updates to ensure optimal network performance.
This documentation explains how to load data from Cisco Meraki to CockroachDB using the open source python library called dlt. CockroachDB is a distributed, cloud-native SQL database that is Kubernetes compatible and free up to 5GB and 1vCPU. By leveraging dlt, users can efficiently transfer and manage their Cisco Meraki data within CockroachDB.
dlt Key Features
- Pipeline Metadata:
dltpipelines leverage metadata to provide governance capabilities, including load IDs that enable incremental transformations and data vaulting. Learn more - Schema Enforcement and Curation: Ensure data consistency and quality by enforcing and curating schemas, which guide the processing and loading of data. Read more
- Schema Evolution:
dltalerts users to schema changes, allowing proactive governance by notifying stakeholders of modifications in the source data’s schema. Learn more - Scaling and Finetuning: Offers several mechanisms and configuration options to scale up and finetune pipelines, such as running extraction, normalization, and load in parallel. Read more
- Supported Data Types:
dltsupports a variety of data types including text, double, bool, timestamp, date, time, bigint, binary, complex, decimal, and wei. Learn more
Getting started with your pipeline locally
dlt-init-openapi0. Prerequisites
dlt and dlt-init-openapi requires Python 3.9 or higher. Additionally, you need to have the pip package manager installed, and we recommend using a virtual environment to manage your dependencies. You can learn more about preparing your computer for dlt in our installation reference.
1. Install dlt and dlt-init-openapi
First you need to install the dlt-init-openapi cli tool.
pip install dlt-init-openapi
The dlt-init-openapi cli is a powerful generator which you can use to turn any OpenAPI spec into a dlt source to ingest data from that api. The quality of the generator source is dependent on how well the API is designed and how accurate the OpenAPI spec you are using is. You may need to make tweaks to the generated code, you can learn more about this here.
# generate pipeline
# NOTE: add_limit adds a global limit, you can remove this later
# NOTE: you will need to select which endpoints to render, you
# can just hit Enter and all will be rendered.
dlt-init-openapi cisco_meraki --url https://raw.githubusercontent.com/dlt-hub/openapi-specs/main/open_api_specs/Business/cisco_meraki.yaml --global-limit 2
cd cisco_meraki_pipeline
# install generated requirements
pip install -r requirements.txt
The last command will install the required dependencies for your pipeline. The dependencies are listed in the requirements.txt:
dlt>=0.4.12
You now have the following folder structure in your project:
cisco_meraki_pipeline/
├── .dlt/
│ ├── config.toml # configs for your pipeline
│ └── secrets.toml # secrets for your pipeline
├── rest_api/ # The rest api verified source
│ └── ...
├── cisco_meraki/
│ └── __init__.py # TODO: possibly tweak this file