Custom

Custom sources allow you to bring in any data and provide your own mapping for it.

We provide a number of pre-defined categories for you to map your data into, these are defined as a set of fields with types.

1. Find the category you'd like to map to

Use the REST API and hit source/category/get to get a listing of all pre-defined categories.

If you don't know which category to use, please use interaction, which can be looked up with the id set to interaction and has ID interaction and version 1

2. Write some IDML to map your data into that category

IDML is a data preparation language designed to process unstructured data at high volumes. The language contains a number of useful functions and operators. Please refer to the IDML documentation for examples of how to use structure and define mappings

3. Create a new Source

You can create one or more "sources" to send data to. A source is a HTTPS endpoint that you can push line-delimited JSON to.
Each source can be configured independently, i.e. it can have a custom IDML mapping, and a custom set of enrichments.

To create new source, send an HTTP POST request to this endpoint:

source/create

with a payload which looks like this

{
  "source_type": "custom",
  "name": "my test api source",
  "resources": [{
    "parameters": {
      "idml": "interaction.content = \"hello world\"",
      "category": "interaction_1"
    }
  }]
}

You can find your category ID and version from the previous API call.

4. Set up a filter and a destination

The data sent to the new sources is immediately processed by the platform. If there's no active filter listening for it, and a configured destination, the data will be dropped.

You can learn about compiling and loading CSDL filters here:

5. Upload data

You can now follow the Ingestion API guide to upload your data.