API Rate Limiting

This page concerns rate limits imposed by DataSift's API to ensure fair usage of the platform.

API Rate Limit Cost

Keep the API Rate Limit Cost in mind:​

  • There is no limit to the number of requests you can send to the Streaming API.
  • All other API requests have a Rate Limit Cost. There's a summary below.

DataSift gives you 10,000 credits and replenishes them every hour. For example, if the Rate Limit Cost of validating a filter is 5 credits and you validate it four times, the total cost is 20 credits, leaving you with a balance of 9,980 credits for the current hour.

If this limit causes your application any problems, please contact DataSift Support.

The DataSift 'hour' starts when you first hit the API. For example, if you compile a rule at 8:05 am and consume all of your 10,000 credits by 8:50 am, DataSift will give you another 10,000 credits at 9:05 am. These credits are free of charge; there is no dollar equivalent. We've documented the Rate Limit Cost for each endpoint in a summary box that looks like this:

Rate limiting is based on the type and the number of requests. You can determine the cost of an API request by inspecting the X-RateLimit-Cost- header.

Response Headers

Every time you make a request to the API we will return the current rate status in the response header. Look for the following keys:

X-RateLimit-Limit: 10000
X-RateLimit-Remaining: 9980
X-RateLimit-Cost: 5
X-RateLimit-Reset-Ttl: 1729
X-RateLimit-Reset: <timestamp>

If the X-RateLimit-Remaining falls to 0, you are rate limited. Wait up to an 1 hour for the next 10,000 credits to reach your account.

If you hit your API limit, all subsequent requests will fail with a suitable error message and HTTP response code of 403.

X-RateLimit-Reset-Ttl gives the number of seconds until the next reset.

X-RateLimit-Reset gives the Unix timestamp when the next reset will occur.

Twitter Rate Limiting

You might also encounter Twitter Rate Limiting. Twitter imposes a limit on the number of Tweets that you can receive in a 24-hour period. At the time of writing that limit is 500,000 Tweets. If you hit your rate limit you will temporarily be unable to continue to receive Twitter data. Please contact our Support desk if you need guidance on this issue.

Twitter applies the rate limit using a rolling 24-hour window (that is, from 'now' back to 24 hours ago).

API Rate Limit Summary

Pylon For Facebook Topic Data
Endpoint Rate Limit Cost
pylon/get 25
pylon/compile 5
pylon/validate 5
pylon/start 25
pylon/stop 5
pylon/update 25
pylon/analyze 25
pylon/sample 25
pylon/tags 25
Account Api
Endpoint Rate Limit Cost
GET /account/identity 5
POST /account/identity 5
GET /account/identity/{id} 5
PUT /account/identity/{id} 5
GET /account/identity/limit/{service} 25
GET /account/identity/{identity_id}/token 5
POST /account/identity/{identity_id}/limit 25
POST /account/identity/{identity_id}/token 25
GET /account/identity/{identity_id}/limit/{service} 25
PUT /account/identity/{identity_id}/limit/{service} 25
GET /account/identity/{identity_id}/token/{service} 5
PUT /account/identity/{identity_id}/token/{service} 5
DELETE /account/identity/{identity_id}/limit/{service} 25
DELETE /account/identity/{identity_id}/token/{service} 5
Core
Endpoint Rate Limit Cost
validate 5
compile 5
usage 25
dpu 5
balance 25
Push
Endpoint Rate Limit Cost
push/get 1
push/validate 5
push/create 5
push/pause 1
push/resume 1
push/update 5
push/stop 1
push/delete 1
push/log 1
pull 0
Managed Sources
Endpoint Rate Limit Cost
source/get 5
source/create 25
source/start 5
source/stop 5
source/delete 25
source/update 25
source/log 5
source/auth/add 25
source/auth/remove 25
source/resource/add 25
source/resource/remove 25
Historics
Endpoint Rate Limit Cost
historics/get 25
historics/prepare 25
historics/start 25
historics/stop 5
historics/pause 5
historics/resume 5
historics/update 5
historics/delete 25
historics/status 5
Historics Preview
Endpoint Rate Limit Cost
preview/get 5
preview/create 25