historics/get

Retrieve details and status codes for one or more Historics queries for a specified user.

An HTTP GET request sent to:

https://api.datasift.com/v1.3/historics/get

A successful call to this endpoint returns: 200 OK plus a JSON object.

Parameters

Parameter Description
id
optional

The id of a Historics query.

If you include an id, DataSift retrieves details of just that single Historics query. If you omit the id, DataSift retrives details of all of your Historics queries and returns the number specified by the max parameter.

Example values: 2490313575797478a5c3

max
optional
default = 20

The maximum number of objects that you want to receive.

The max parameter is optional. If you omit it and you do no specify a Historics id, DataSift retrieves details of the first 20 Historics queries it finds.

If you set max at 50, for example, and leave the page parameter blank or set page to 1, DataSift returns the first 50 objects it finds.

If you set max to 50 and set page to 3, for example, DataSift assumes that each 'page' of results contains 50 objects; it discards the first 2 pages that it finds (in other words, it discards 100 objects) and gives you the next 50 objects. If each object were numbered, this would mean you receive objects 101 to 150.

Example values: 42

page
optional
default = 1

The page number.

Example values: 2

with_estimate
optional
default = 1

Include an estimate of the completion time in the JSON output. It appears as the estimated_completion element.

You must include this parameter and set it to 1 to receive job completion time estimate for the whole query and each chunk. The values of the estimated_completion keys are UTC timestamps of the estimated completion time for the whole job and for each individual chunk.

Example values: 0

Examples

You can call /v1.3/historics/get with or without the Historics id. The following examples discuss what you should expect when these requests are successful.

  1. Here is an example of a call to /v1.3/historics/get that includes a Historics id:

    curl -X GET http://api.datasift.com/v1.3/historics/get?id=historics-id 
            -H 'Authorization:username:apikey'

    A successful call to should result in the following response that includes information about the status of the whole query and of its chunks:

    {
            "id": "9e97d9ac115e58fcb7ab",
            "definition_id": "e4d0b9bb266e8f3f2de405f6c1a2de23",
            "name": "historics-get-test",
            "start": 1325376001,
            "end": 1325894401,
            "created_at": 1363271961,
            "status": "running",
            "progress": 15,
            "sources": [
                "twitter"
            ],
            "sample": 100,
            "chunks": [
                {
                    "status": "succeeded",
                    "progress": 100,
                    "start_time": 1325376001,
                    "end_time": 1325462400
                },
                {
                    "status": "running",
                    "progress": 5,
                    "start_time": 1325462400,
                    "end_time": 1325548800
                },
                {
                    "status": "init",
                    "progress": 0,
                    "start_time": 1325548800,
                    "end_time": 1325635200
                },
                {
                    "status": "init",
                    "progress": 0,
                    "start_time": 1325635200,
                    "end_time": 1325721600
                },
                {
                    "status": "init",
                    "progress": 0,
                    "start_time": 1325721600,
                    "end_time": 1325808000
                },
                {
                    "status": "init",
                    "progress": 0,
                    "start_time": 1325808000,
                    "end_time": 1325894400
                },
                {
                    "status": "init",
                    "progress": 0,
                    "start_time": 1325894400,
                    "end_time": 1325894401
                }
            ]
        }

    Property: Description:
    id

    The Historics id, a unique identifier for this Historics query.

    definition_id The hash of the CSDL for this Historics query.
    name The name of this Historics query. You specified this in your call to /historics/prepare.
    start

    Unix timestamp defining the start of a day in our Historics archive. A day starts at midnight and ends one second before midnight 24 hours later.

    In your call to /historics/prepare, you can specify any start time to one second of accuracy. However, the 'start' returned in the JSON is always midnight on the day defined by the start value you supplied in your API call. The status element in the JSON (see below) gives you the percentage data coverage available in the archive for the interval between the JSON start and end.

    end

    Unix timestamp defining the end of a day in our Historics archive. A day starts at midnight and ends one second before midnight 24 hours later.

    In your call to /historics/prepare, you can specify any end time to one second of accuracy. However, the 'end' returned in the JSON is always one second before midnight on the day defined by the end value you supplied in your API call. The status element in the JSON (see below) gives you the percentage data coverage available in the archive for the interval between the JSON start and end.

    We frequently find that in calls to /historics/prepare, people specify their start to be midnight on one day and their end to be midnight on another day. Technically, this means that their request ends one second after the end of a day as defined in our archive. We overlook this second; it makes no sense to consider data coverage for an additional day just for the sake of one extra second.

    created_at The moment when you hit /historics/prepare for this Historics query.
    status

    The parent status element contains the current status of this Historics query. Status elements insode chunk objects contain the current status of a Historics query chunk.

    progress

    At the top level in the JSON, progress indicates the percentage completion of the entire Historics query. There is a pregress indicator for each chunk too. See below.

    sources An array of the data sources you selected for this Historics query.
    sample The value you selected in the sample parameter when you called /historics/prepare for this query.
    chunks Details for each 'chunk' of the job. Take a look at Understanding Chunks.
    progress At the lower level in the JSON, progress indicates the percentage completion of each chunk of a Historics query. There is a pregress indicator for the entire Historics query too. See above.
    start_time The start timestamp of a chunk.
    end_time The end timestamp of a chunk.
    estimated_completion A Unix timestamp that indicates when we expect to finish processing this query.
    delivery_count When this appears inside any chunk (see example above) it is the number of interactions delivered in that chunk. When it appears outside the array of chunks, it is the sum of all the individual delivery_count values. If the progress indicator is 0, the delivery_count is 0. The delivery_count is relevant when the status is: running, paused, killed, delayed, stopped, or succeeded.
  2. Here is an example of a call to /v1.3/historics/get that includes a Historics id and the with_estimate parameter:

    curl -X GET http://api.datasift.com/v1.3/historics/get?id=historics-id&with_estimate=1 
            -H 'Authorization:username:apikey'

    The output includes estimated_completion keys for the whole job and for the individual chunks.

    {
            "id": "9e97d9ac115e58fcb7ab",
            "definition_id": "cdd0b9bb266e8f3f2de405f6c1a2b23e",
            "name": "job-estimation-test",
            "start": 1325376001,
            "end": 1325894401,
            "created_at": 1363271961,
            "status": "running",
            "progress": 15,
            "sources": [
                "twitter"
            ],
            "sample": 100,
            "delivery_count": 300,
            "chunks": [
                {
                    "status": "succeeded",
                    "progress": 100,
                    "start_time": 1325376001,
                    "end_time": 1325462400,
                    "estimated_completion": 1363274434,
                    "delivery_count": 200
                },
                {
                    "status": "running",
                    "progress": 5,
                    "start_time": 1325462400,
                    "end_time": 1325548800,
                    "estimated_completion": 1363277434,
                    "delivery_count": 100
                },
                {
                    "status": "init",
                    "progress": 0,
                    "start_time": 1325548800,
                    "end_time": 1325635200,
                    "estimated_completion": 1363285834,
                    "delivery_count": 0
                },
                {
                    "status": "init",
                    "progress": 0,
                    "start_time": 1325635200,
                    "end_time": 1325721600,
                    "estimated_completion": 1363279834,
                    "delivery_count": 0
                },
                {
                    "status": "init",
                    "progress": 0,
                    "start_time": 1325721600,
                    "end_time": 1325808000,
                    "estimated_completion": 1363282234,
                    "delivery_count": 0
                },
                {
                    "status": "init",
                    "progress": 0,
                    "start_time": 1325808000,
                    "end_time": 1325894400,
                    "estimated_completion": 1363284034,
                    "delivery_count": 0
                },
                {
                    "status": "init",
                    "progress": 0,
                    "start_time": 1325894400,
                    "end_time": 1325894401,
                    "estimated_completion": 1363287634,
                    "delivery_count": 0
                }
            ],
            "estimated_completion": 1363287634
        }

  3. If you do not provide a Historics id and you have created 79 Historics queries and you set max to 2 in your API call:

    curl -X GET http://api.datasift.com/v1.3/historics/get?id=historics-id&max=2 
            -H 'Authorization:username:apikey'

    Notice that information about chunks is not included. To receive chunk information, you must specify a Historics id.

    {
            "data": [
                {
                    "id": "6b42f539793a02bfd68c",
                    "definition_id": "6beacf0faeb77c974ab801733ed2c597",
                    "name": "twitter",
                    "start": 1274742000,
                    "end": 1339110000,
                    "created_at": 1339147632,
                    "status": "init",
                    "progress": 0,
                    "sources": [
                        "facebook",
                        "twitter"
                    ],
                    "sample": 100
                },
                {
                    "id": "fdf6652ac82206358eab",
                    "definition_id": "93558e17de15072fa126370c37c5bd8f",
                    "name": "test",
                    "start": 1337040000,
                    "end": 1337126400,
                    "created_at": 1337161927,
                    "status": "succeeded",
                    "progress": 100,
                    "sources": [
                        ""
                    ],
                    "sample": 100
                }
            ],
            "count": 79
        }

Notes

  1. Be sure to read our introduction to the Historics API to get started with these endpoints.
  2. All calls to the API must be properly authenticated with a DataSift username and API key.
  3. All calls to the API must be versioned. The current version is v1.3.

Revision history

v1.1

The volume_info field no longer appears in the JSON output.

The with_estimate field in the JSON output now appears when you do not specify the optional id parameter. Prior to v1.1 it appeared only if you specified an id.

From v1.1, each chunk returned in the JSON output has a delivery_count against it. The main object has a delivery_count showing the sum of the delivery_count values for all the chunks.

Resource information

Rate limit cost: 25

Requires authentication: Yes

Response formats: JSON, JSONP