Pipe Concepts

This page contains additional information about pipes.

Pipe types

Single provider pipes

Many pipes use a single provider for processing. Providers can be a third-party service like LeadMagic  or ZeroBounce . Native pipes use Pipe0 as a provider.

Waterfall pipes

Waterfall pipes use many providers in a cascading sequence. We try provider A. If the data is found, it is returned. If the data is not found we try provider B. If the data is not found again, we try provider C and so on.

A billing event occurs for the provider that returns a result. Providers that return no_result are not billed.

Waterfall enrichments can drastically increase data coverage. But keep in mind that trying many providers can slow down processing speed.

Choosing providers

Waterfall pipes have a config.providers option. The option is typically of the type Array<{ provider: <PROVIDER_NAME> }>. If no config is provided the default config is applied. If you remove a provider from the array, this provider is removed from your processing sequence.

Providers are processed according to their position in the config.providers array. The provider with the index 0 is processed first. You are encouraged to change the order.

Field mode

The field mode determines the shape of the config object a pipe takes.

Field mode: static

Most pipes have static input and output fields. When field_mode is set to static the pipe’s in- and outputs are known in advance.

You can point the input fields to different fields or rename the outputs but the pipe’s API is clearly defined in advance.

Example:

When using the pipe people:workemail:waterfall@1, we use the input fields name and company_website_url to generate the output field work_email.

Field mode: prompt

The user is required to specify the inputs and outputs of a pipe via a prompt.

Example:

The pipe run:prompt@1 uses any number of input fields to generate any number of output fields. The user may specify any input or output field by using prompt tags.

Prompt tags

To define fields in prompt mode, use the following syntax to define fields.

{
  "prompt": "Based on the name of the person {{ input name type=\"string\" required=\"true\" }} assign a character from the TV show \"The Simpsons\" to that person. {{ output value description=\"Name of a character from the TV show 'The Simpsons\'\" }}",
}

type: “string” | “number” | “boolean”
required: “true” | “false”
description: string

Pipeline

Pipeline is the internal data model we use to enrich data. A pipeline consists of:

(1) Pipes and their order (2) Fields added by your input (3) Output fields added by pipes (4) Logic, conditions, and actions.

Pipeline validation

For a pipeline to be valid the following conditions have to be true:

1. The input requirements of each pipe must be met incl. field format.
2. Your pipeline cannot contain self-referencing pipes (the input of a pipe points to its output)

Pipeline requests

Internally, we convert the request object into a pipeline.

Pipeline response

After pipeline validation and processing, we convert a pipelines into the response objects.