historics/status

Check data coverage in the archive for a specified interval.

An HTTP GET request sent to:

https://api.datasift.com/v1.3/historics/status

A successful call to this endpoint returns: 200 OK plus a JSON object.

Parameters

Parameter Description
strart
required

Unix timestamp for the start time.

Example values: 1325548800

end
required

Unix timestamp for the end time.

Example values: 1325549000

sources
optional

Comma separated list of data sources to include. Currently, the only value you can use is twitter. In the future, you will be able to choose any source listed as a valid value that you would use in the interaction.type target.

Example values: twitter

Examples

  1. Suppose that you want to run a Historics query on Twitter and Facebook data for January 1, 2012. You can hit the /historics/status endpoint to check how much data the DataSift archive holds for that day. First, find the start and end timestamps; they are 1325376000 and 1325462399. Then send this API request:

    curl -X GET https://api.datasift.com/v1.3/historics/status?start=1325376000&end=1325462399&sources=twitter,facebook 
            -H 'Authorization:username:apikey'

    DataSift responds with information in this format:

    [
            {
                "start": "1325376000",
                "end": 1325462400,
                "sources": {
                    "facebook": {
                        "augmentations": {
                            "demographic": 100,
                            "language": 100,
                            "links": 100,
                            "salience": 100
                        },
                        "versions": [
                            "1"
                        ],
                        "status": 100
                    },
                    "twitter": {
                        "augmentations": {
                            "demographic": 100,
                            "klout": 100,
                            "language": 100,
                            "links": 100,
                            "salience": 100,
                            "trends": 100
                        },
                        "versions": [
                            "1"
                        ],
                        "status": 100
                    }
                }
            }
        ]

    Property: Description:
    start

    Unix timestamp defining the start of a day in our Historics archive. A day starts at midnight and ends one second before midnight 24 hours later.

    In your call to /historics/prepare, you can specify any start time to one second of accuracy. However, the 'start' returned in the JSON is always midnight on the day defined by the start value you supplied in your API call. The status element in the JSON (see below) gives you the percentage data coverage available in the archive for the interval between the JSON start and end.

    end

    Unix timestamp defining the end of a day in our Historics archive. A day starts at midnight and ends one second before midnight 24 hours later.

    In your call to /historics/prepare, you can specify any end time to one second of accuracy. However, the 'end' returned in the JSON is always one second before midnight on the day defined by the end value you supplied in your API call. The status element in the JSON (see below) gives you the percentage data coverage available in the archive for the interval between the JSON start and end.

    We frequently find that in calls to /historics/prepare, people specify their start to be midnight on one day and their end to be midnight on another day. Technically, this means that their request ends one second after the end of a day as defined in our archive. We overlook this second; it makes no sense to consider data coverage for an additional day just for the sake of one extra second.

    status The status element is based on an examination of the archive in the time interval between the start and end timestamps in the JSON (see above). For each data source, it shows the percentage of data coverage available in the interval.
    versions This element is reserved for future use.
    links For each Augmentation (Klout, Links, and so on) the JSON shows the percentage of coverage available for the time interval defined in start and end.
  2. If your request covers data for two days, you will receive two blocks in your JSON output:

    curl -X GET https://api.datasift.com/v1.3/historics/status?start=1325376000&end=1325548799&sources=twitter 
            -H 'Authorization:username:apikey'

    DataSift responds with this JSON:

    [
            {
                "start": "1325376000",
                "end": 1325462400,
                "sources": {
                    "twitter": {
                        "augmentations": {
                            "demographic": 100,
                            "klout": 100,
                            "language": 100,
                            "links": 100,
                            "salience": 100,
                            "trends": 100
                        },
                        "versions": [
                            "1"
                        ],
                        "status": 100
                    }
                }
            },
            {
                "start": "1325462400",
                "end": 1325548800,
                "sources": {
                    "twitter": {
                        "augmentations": {
                            "demographic": 100,
                            "klout": 100,
                            "language": 100,
                            "links": 100,
                            "salience": 100,
                            "trends": 100
                        },
                        "versions": [
                            "1"
                        ],
                        "status": 100
                    }
                }
            }
        ]

Checking availability

It's a good idea to run the /historics/status endpoint before you launch a Historics query. It allows you to determine how much data is available for the time interval you're looking at for your chosen data sources.

This is useful because it saves you the time and expense of running a Historics query that might return little data.

If you omit the sources parameter, DataSift returns information for all the data sources available.

Understanding the JSON output

The endpoint returns a JSON object that shows you the percentage coverage that DataSift has for each data source that you selected and each augmentation that it can take.

In the first example below, the call looks at our archived data for Twitter and Facebook for a specified time interval. The JSON shows the status for Twitter is 99 which indicates that we have 99% coverage. Similarly, the augmentations element tells us that we have 50% coverage for the Klout augmentation for Twitter during that interval and 100% coverage for the Links augmentation.

Notes

  1. Be sure to read our introduction to the Historics API to get started with these endpoints.
  2. All calls to the API must be properly authenticated with a DataSift username and API key.
  3. All calls to the API must be versioned. The current version is v1.3.

Resource information

Rate limit cost: 5

Requires authentication: Yes

Response formats: JSON, JSONP