Ingestion API

The ingestion API offers a way to send data to DataSift. With this API you can ingest data in real time, it's designed to be used when you want to bring your own data to be augmented and filtered by the streaming platform.

You can interact with the ingestion API via HTTP POST requests.

Authentication

Please consult our API authentication page for details on how to authorize API calls.

How it Works

Before using the Ingestion API, you must create an ingestion source using Managed Sources, either using DataSift's website UI or the source/create API endpoint in the REST API.

Sending a single document

To send a single document, create an HTTP POST request with the JSON document as the body, and the Content-Type header set to application/json, and send it to https://in.datasift.com/SOURCEID, where SOURCEID is the ID for the Source created in the previous step.

Example

curl -H 'Authorization: andi.miller:myapikey' -H 'Content-Type: application/json' -X POST "https://in.datasift.com/mysourceid" --data '{"some":"value"}'

{"accepted":1,"total_message_bytes":17}

Sending a batch

It's also possible to send multiple documents at once, by posting them to the same HTTP endpoint (https://in.datasift.com/SOURCEID`), where the request payload consists in multiple JSON documents, separated by new lines.

Example

body.json

{}
{"hello": "world"}
{}

curl -H 'Authorization: andi.miller:myapikey' -H 'Content-Type: application/json' -X POST "https://in.datasift.com/d315e303a7914b8b980755    931c812680" --data-binary @body.json

{"accepted":3,"total_message_bytes":22}

Endpoints

For HTTP POST ingestion, the endpoint is:

https://in.datasift.com/

Rate Limits

There are 3 rate limits which can be set by your account manager, these default to:

  • Max concurrent connections per user: 2
  • Max uploaded bytes per minute: 614400
  • Max HTTP POSTs per minute: 10000