Planning Your Migration from API v1.2 to v1.3

Richard Caudle | 2nd March 2016

Today we released version 1.7 of PYLON for Facebook Topic Data. This release introduces version 1.3 of the DataSift API. For PYLON customers this means significant improvement to features but also code changes to move to the new API. In this post we'll take a look at the changes and cover points you'll need to consider when making the move.

Note that version 1.2 of the API will be available for a while yet. Check out the API changelog for full details of planned API deprecations.

Even if you are not a PYLON customer we recommend you switch to use API 1.3 (even though there are no changes outside of the PYLON features) as API versions will be deprecated over time.

Introducing PYLON filter swapping

We've introduced a new version of the API to support the new filter swapping feature for PYLON.

This new feature allows you to update the CSDL for your interaction filters in real time whilst your recordings are running. The new version of your filter will be applied in seconds. Data will continue to be recorded for your previous filter version until the new filter has been applied, so there will be no gap in your recording.

To support this feature recordings (and their indexes) are now referenced by id rather than the hash for the CSDL definition. PYLON API endpoints have been updated to reflect this.

Take a look at the updated endpoints >

API v1.3 workflow

For version v1.2 of the API your workflow was:

  • Define your interaction filter in CSDL.
  • Compile your CSDL definition using pylon/compile to receive a hash.
  • Start a recording using pylon/start providing the hash and a name for your recording.
  • Submit analysis queries using pylon/analyze providing the hash for the recording you want to analyze.
  • Stop the recording when applicable using pylon/stop supplying the hash for the recording.

For API v1.3 this now changes to use an id as the identifier for the recording:

  • Define your interaction filter in CSDL.
  • Compile your CSDL definition using pylon/compile to receive a hash.
  • Start a recording using pylon/start providing the hash and a name for your recording.
    • The API response will include the id for your recording.
  • Submit analysis queries using pylon/analyze providing the id for the recording you want to analyze.
  • You can update the definition for your interaction filter, even whilst it is running using pylon/update supplying the id for the recording and a new hash for the CSDL definition.
  • Stop the recording when applicable using pylon/stop supplying the id for the recording.

Essentially the workflow remains the same, but you need to switch to using the id supplied by pylon/start to reference your recordings. Of course now you have the option to update your filter on-the-fly!

What about my existing recordings?

The first question you might have is 'What about my existing recordings?'.

For existing recordings in your account we have backfilled ids.

When you migrate to the new API you will need to query the API for the ids for your existing recordings and add these to your database (or however you maintain your list of recordings for your customers).

You can find the IDs for your recordings using the pylon/get endpoint.

For example the following request:

curl -H "Auth: username:apikey" http://api.datasift.com/v1.3/pylon/get

Will return a response such as:

{
  "count": 14,
  "page": 1,
  "pages": 1,
  "per_page": 25,
  "subscriptions": [
    {
      "id": "12231f2f3825fe4c79e4f5def24c41fa8912d199",
      "volume": 101231,
      "start": 1453482165,
      "end": 1453482171,
      "status": "running",
      "name": "Example recording",
      "reached_capacity": false,
      "identity_id": "ss3fd831cb9f1441beeae6c09b5af15f",
      "hash": "bf6dd0ad87e911bb72c3f6ee4750247r",
      "remaining_index_capacity": 1000000,
      "remaining_account_capacity": 1000000
    },
  ...
}

Can I continue to use the current API version?

Yes. You can continue to use v1.2 of the API for some time yet, allowing you to plan your migration. Check out the API changelog for full details of planned API deprecations.

Whilst you continue to use v1.2 you cannot make use of the new filter swapping feature as v1.2 does not support this feature.

If you continue to use v1.2 of the API any recordings you create will be given an id behind the scenes. When you switch to v1.3 of the API any recordings you have created will have ids that you can use going forwards.

Note that once you start creating recordings using the v1.3 API you cannot reference these using v1.2 as they are identified by ids not hashes.

What about client libraries?

We have published new builds of each of our client libraries all which now operate with the new API version.

When you upgrade to one of the latest libraries you will automatically be using API v1.3 and move away from 1.2.


Previous post: Announcing PYLON 1.7 - Introducing Interaction Filter Swapping

Next post: Filter Swapping (part 1)