Skip to main content
Version: devel

Zendesk

Need help deploying these sources or figuring out how to run them in your data stack?
Join our Slack community or Book a call with a dltHub Solutions Engineer.

Zendesk is a cloud-based customer service and support platform. It offers a range of features, including ticket management, self-service options, knowledge base management, live chat, customer analytics, and talks.

This Zendesk dlt verified source and pipeline example loads data using the “Zendesk Support API”, "Zendesk Chat API", and "Zendesk Talk API" to the destination of your choice.

Endpoints that can be loaded using this verified source are:

NameDescription
SUPPORT_ENDPOINTS"users", "sla_policies", "groups", "organizations", "brands"
SUPPORT_EXTRA_ENDPOINTS"activities", "automations", "custom_agent_roles", "dynamic_content", "group memberships",
"job_status", "macros", "organization_fields", "organization memberships", "recipient_addresses",
"requests", "satisfaction_ratings", "sharing_agreements", "skips", "suspended_tickets", "targets",
"ticket_forms", "ticket_metrics", "triggers", "user_fields", "views", "tags"
TALK_ENDPOINTS"calls", "addresses", "greeting_categories", "greetings", "ivrs",
"phone_numbers", "settings", "lines", "agents_activity"
INCREMENTAL_TALK_ENDPOINTS"calls", "logs"

To get the complete list of endpoints, please refer to "zendesk/settings.py".

Setup guide​

Grab credentials​

You can load data from three types of Zendesk services, which are:

  • Zendesk Support
  • Zendesk Chat
  • Zendesk Talk

Zendesk support​

Zendesk support can be authenticated using one of the following three methods:

The simplest way to authenticate is via subdomain + email address + password, since these details are already available and you don't have to generate any tokens. Alternatively, you can also use API tokens or OAuth tokens.

Grab subdomain​

  1. Log into Zendesk to find your subdomain in the URL. E.g., for https://www.dlthub.zendesk.com, the subdomain is "dlthub".

Grab Zendesk support API token​

  1. In Zendesk (top right), select Admin Center.
  2. Choose "Apps and Integrations".
  3. Navigate to APIs and select Zendesk API. Activate “Password access” & “Token access”.
  4. Click “Add API token”, add a description, and note down the API token.
  5. The token displays just once; note it safely.

Grab Zendesk support OAuth token​

To get the OAuth token, see the documentation. Here's a summarized version:

  1. Obtain client ID via Zendesk API: send this curl request and note the response's client ID.

  2. Alternatively, fetch client ID via OAuth using this.

  3. To get the full token using the client ID obtained above, you can follow the instructions here.

     curl https://{subdomain}.zendesk.com/api/v2/oauth/tokens.json \
    -X POST \
    -v -u {email_address}:{password} \
    -H "Content-Type: application/json" \
    -d '{
      "token": {
        "client_id": 223443,
        "scopes": [
          "read"
        ]
      }
    }'

    We've set the scope as 'read', but you can customize the scope as needed.

  4. In response to the above request, you'll get a full token which can be used to configure Zendesk support.

Zendesk chat​

Zendesk chat can be authenticated using this method:

  • Method 1 (subdomain + Chat OAuth token)

    Note: OAuth tokens for Zendesk chat and support differ, requiring separate generation procedures.

Subdomain​

Log into Zendesk to find your subdomain in the URL. E.g., for https://www.dlthub.zendesk.com, the subdomain is "dlthub".

Grab Zendesk chat OAuth token​

To generate a Zendesk chat OAuth token, please refer to this documentation . Below is a summary of the steps:

  1. Access Zendesk Chat directly or through the top right "Chat" option in Zendesk product.
  2. Navigate to "Settings" > "Account" > "API" > "Add API client".
  3. Fill in the client name, company, and redirect URLs (default: http://localhost:8080).
  4. Record the "CLIENT_ID" and "SUBDOMAIN".
  5. Format the below URL with your own CLIENT_ID and SUBDOMAIN, paste it into a new browser tab, and press Enter.
    https://www.zopim.com/oauth2/authorizations/new?response_type=token&client_id=CLIENT_ID&scope=read%20write&subdomain=SUBDOMAIN
  6. The call will be made, possibly asking you to log in and select 'Allow' to generate the token.
  7. If the call succeeds, your browser's address field will contain your new OAuth token (returned as the access_token value).
  8. Despite the seeming error message displayed in the browser's main window, if 'access_token' is returned in the browser's URL field then it worked! Zendesk Chat
  9. Safely store the OAuth token to authenticate Zendesk Chat for retrieving data.
  10. There are several other methods to obtain a Zendesk chat token as given in the full documentation here.

Zendesk talk​

Zendesk Talk fetches the data using the Zendesk Talk API.

  1. Obtaining credentials for Zendesk Talk mirrors the process for Zendesk support.
  2. Use existing Zendesk support credentials or create new ones.

Note: The Zendesk UI, which is described here, might change. The full guide is available at this link.

Initialize the verified source​

To get started with your data pipeline, follow these steps:

  1. Enter the following command:

    dlt init zendesk duckdb

    This command will initialize the pipeline example with Zendesk as the source and duckdb as the destination.

  2. If you'd like to use a different destination, simply replace duckdb with the name of your preferred destination.

  3. After running this command, a new directory will be created with the necessary files and configuration settings to get started.

For more information, read the guide on how to add a verified source..

Add credentials​

  1. In the .dlt folder, there's a file called secrets.toml. It's where you store sensitive information securely, like access tokens. Keep this file safe. Here's its format for service account authentication:

    #Zendesk support credentials
    [sources.zendesk.credentials]
    subdomain = "subdomain" # Zendesk subdomain
    email = "set me up" # Email used to login to Zendesk
    password = "set me up" # Password for Zendesk account
    token = "set me up" # For API token auth
    oauth_token = "set me up" # Use Zendesk support OAuth token or Zendesk chat OAuth token

    For data retrieval from Zendesk Support or Talk, choose one of the following verification methods:

    Note: Use the Zendesk Support OAuth token for configuring Zendesk Support, and for Chat, utilize the OAuth token specific to Zendesk Chat.

  2. Finally, enter credentials for your chosen destination as per the docs.

For more information, read the General Usage: Credentials.

Run the pipeline​

  1. Before running the pipeline, ensure that you have installed all the necessary dependencies by running the command:

    pip install -r requirements.txt

  2. You're now ready to run the pipeline! To get started, run the following command:

    python zendesk_pipeline.py

  3. Once the pipeline has finished running, you can verify that everything loaded correctly by using the following command:

    dlt pipeline <pipeline_name> show

    For example, the pipeline_name for the above pipeline example is dlt_zendesk_pipeline, you may also use any custom name instead.

For more information, read the guide on how to run a pipeline.

Sources and resources​

dlt works on the principle of sources and resources.

Source zendesk_talk​

This function retrieves data from Zendesk Talk for phone calls and voicemails.

@dlt.source(max_table_nesting=2)
def zendesk_talk(
    credentials: TZendeskCredentials = dlt.secrets.value,
    start_date: Optional[TAnyDateTime] = START_DATE,
    end_date: Optional[TAnyDateTime] = None,
) -> Iterable[DltResource]:
   ...

credentials: Authentication credentials.

start_date: Start time for data range, defaults to January 1st, 2000.

end_date: End time for data range. If not provided, only new data is retrieved after the initial run.

Resource talk_resource​

This function loads data from the Zendesk Talk endpoint.

def talk_resource(
    zendesk_client: ZendeskAPIClient,
    talk_endpoint_name: str,
    talk_endpoint: str,
    pagination_type: PaginationType,
) -> Iterator[TDataItem]:
   ...

zendesk_client: An instance of ZendeskAPIClient for making API calls to Zendesk Talk.

talk_endpoint_name: The name of the talk endpoint.

talk_endpoint: The actual URL ending of the endpoint.

pagination_type: Type of pagination used by the endpoint.

Other functions similar to the source zendesk_talk and resources similar to talk_endpoint are:

Function NameTypeDescription
zendesk_chatsourceRetrieves data from Zendesk Chat for chat interactions
chats_table_resourceresourceRetrieves chats from Zendesk
talk_incremental_resourceresourceRetrieves data incrementally from a Zendesk Talk endpoint.
zendesk_supportsourceRetrieves data from Zendesk Support for tickets, users, brands, organizations, and groups
ticket_eventsresourceRetrieves records of all changes made to a ticket, including state, etc.
ticketsresourceRetrieves the data for the ticket table, which can be pivoted and columns renamed
ticket_metric_eventsresourceRetrieves ticket metric events from the start date, defaulting to January 1st of the current year
basic_resourceresourceRetrieves basic loader for Zenpy endpoints with pagination support

Customization​

Create your own pipeline​

If you wish to create your own pipelines, you can leverage source and resource methods from this verified source.

  1. Configure the pipeline by specifying the pipeline name, destination, and dataset as follows:

    pipeline = dlt.pipeline(
        pipeline_name="dlt_zendesk_pipeline",  # Use a custom name if desired
        destination="duckdb",  # Choose the appropriate destination (e.g., duckdb, redshift, post)
        dataset_name="sample_zendesk_data"  # Use a custom name if desired
    )

  2. To load data related to support, talk, and chat:

     # Zendesk support source function
     data_support = zendesk_support(load_all=True)
     # Zendesk chat source function
     data_chat = zendesk_chat()
     # Zendesk talk source function
     data_talk = zendesk_talk()
     # Run pipeline with all 3 sources
     info = pipeline.run([data_support, data_chat, data_talk])
     print(info)

  3. To load data related to support, chat, and talk in incremental mode:

    pipeline = dlt.pipeline(
         pipeline_name="dlt_zendesk_pipeline",  # Use a custom name if desired
         destination="duckdb",  # Choose the appropriate destination (e.g., duckdb, redshift, post)
         dev_mode=False,
         dataset_name="sample_zendesk_data"  # Use a custom name if desired
    )
    data = zendesk_support(load_all=True, start_date=START_DATE)
    data_chat = zendesk_chat(start_date=START_DATE)
    data_talk = zendesk_talk(start_date=START_DATE)
    info = pipeline.run(data=[data, data_chat, data_talk])
    print(info)

    Supports incremental loading for Support, Chat, and Talk endpoints. By default, it fetches data from the last load time in the dlt state or from 1st Jan 2000 if no prior load. This approach ensures data retrieval since the specified date, while still updating the last load time.

  4. To load historical data in weekly ranges from Jan 1st, 2023, then switch to incremental loading for new tickets.

     # Load ranges of dates between January 1st, 2023, and today
     min_start_date = pendulum.DateTime(year=2023, month=1, day=1).in_timezone("UTC")
     max_end_date = pendulum.today()
     # Generate tuples of date ranges, each with 1 week in between.
     ranges = make_date_ranges(min_start_date, max_end_date, datetime.timedelta(weeks=1))

     # Run the pipeline in a loop for each 1-week range
     for start, end in ranges:
         print(f"Loading tickets between {start} and {end}")
         data = zendesk_support(start_date=start, end_date=end).with_resources("tickets")
         info = pipeline.run(data=data)
         print(info)

     # Backloading is done, now we continue loading with incremental state, starting where the backloading left off
     print(f"Loading with incremental state, starting at {end}")
     data = zendesk_support(start_date=end).with_resources("tickets")
     info = pipeline.run(data)
     print(info)

    This can be useful to reduce the potential failure window when loading large amounts of historic data. This approach can be used with all incremental Zendesk sources.

Additional Setup guides​

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.