API Rate Limiting

This page describes rate limits imposed by DataSift's API to ensure fair usage of the platform.

PYLON analysis limits

Each PYLON-based product sets specific limits for the analysis you can request.

See the product documentation for details of these limits:

General API rate limit

Regardless of the product you are using, any calls to the REST API are subject to the general API rate limit.

DataSift gives you 10,000 credits and replenishes them every hour. For example, if the Rate Limit Cost of validating a filter is 5 credits and you validate it four times, the total cost is 20 credits, leaving you with a balance of 9,980 credits for the current hour.

The table below summarizes the cost of each REST API endpoint.

If this limit causes your application any problems, please contact DataSift Support.

The DataSift 'hour' starts when you first hit the API. For example, if you compile a rule at 8:05 am and consume all of your 10,000 credits by 8:50 am, DataSift will give you another 10,000 credits at 9:05 am. These credits are free of charge; there is no dollar equivalent. We've documented the Rate Limit Cost for each endpoint in a summary box that looks like this:

Rate limiting is based on the type and the number of requests. You can determine the cost of an API request by inspecting the X-RateLimit-Cost header.

Response Headers

Every time you make a request to the API we will return the current rate status in the response header. Look for the following keys:

X-RateLimit-Limit: 10000
X-RateLimit-Remaining: 9980
X-RateLimit-Cost: 5
X-RateLimit-Reset-Ttl: 1729
X-RateLimit-Reset: <timestamp>

If the X-RateLimit-Remaining falls to 0, you are rate limited. Wait up to an 1 hour for the next 10,000 credits to reach your account.

If you hit your API limit, all subsequent requests will fail with a suitable error message and HTTP response code of 403 or 429.

X-RateLimit-Reset-Ttl gives the number of seconds until the next reset.

X-RateLimit-Reset gives the Unix timestamp when the next reset will occur.

API Rate Limit Summary

Pylon Asynchronous Analysis
Endpoint Rate Limit Cost
GET /pylon/{service}/task 5
GET /pylon/{service}/task/{type} 5
GET /pylon/{service}/task/{id} 5
GET /pylon/{service}/task/{type}/{id} 5
POST /pylon/{service}/task 25
Reference Data Api
Endpoint Rate Limit Cost
GET /pylon/{service}/reference 5
GET /pylon/{service}/reference/{slug} 5
Pylon Private Recordings
Endpoint Rate Limit Cost
pylon/analyze 25
pylon/validate 5
pylon/sample 25
pylon/start 25
pylon/compile 5
pylon/get 25
pylon/stop 5
pylon/tags 25
pylon/update 25
Account Api
Endpoint Rate Limit Cost
GET /account/identity 5
POST /account/identity 5
GET /account/identity/{id} 5
PUT /account/identity/{id} 5
GET /account/identity/limit/{service} 25
GET /account/identity/{identity_id}/token 5
POST /account/identity/{identity_id}/limit 25
POST /account/identity/{identity_id}/token 25
GET /account/identity/{identity_id}/limit/{service} 25
PUT /account/identity/{identity_id}/limit/{service} 25
GET /account/identity/{identity_id}/token/{service} 5
PUT /account/identity/{identity_id}/token/{service} 5
DELETE /account/identity/{identity_id}/limit/{service} 25
DELETE /account/identity/{identity_id}/token/{service} 5
GET /account/usage 5
Endpoint Rate Limit Cost
validate 5
compile 5
usage 25
dpu 5
balance 25
Endpoint Rate Limit Cost
push/get 1
push/validate 5
push/create 5
push/pause 1
push/resume 1
push/update 5
push/stop 1
push/delete 1
push/log 1
pull 0
Managed Sources
Endpoint Rate Limit Cost
source/get 5
source/create 25
source/start 5
source/stop 5
source/delete 25
source/update 25
source/log 5
source/auth/add 25
source/auth/remove 25
source/resource/add 25
source/resource/remove 25
Endpoint Rate Limit Cost
historics/get 25
historics/prepare 25
historics/start 25
historics/stop 5
historics/pause 5
historics/resume 5
historics/update 5
historics/delete 25
Historics Preview
Endpoint Rate Limit Cost
preview/get 5
preview/create 25