DataSift is very simple to use. Build filters to find the information you want in real time.

1. The filters are written in our programming language. It's called the Curated Stream Definition Language (CSDL).

2. The key feature of CSDL is its set of operators.

3. The filters take information from social media sites and augmentations. Together these are called targets.

4. You can write applications that access DataSift through our APIs.

5. You can test your filters in our user interface before you start to explore the APIs.

6. Please read our Understanding Billing guide and Billing FAQ.

7. Please read Targets vs Output Data to understand the difference between filtering data and consuming data.

We've rewritten this page but the earlier version is still available if you need it.

Are you trying to set up Push or Historics? You're in the right place. Read this page and then go to Client Library: Push and Historics.

The best way to understand how our client libraries work is to take a high-level view to understand the general concepts and then dip down into your language of choice. All the client libraries follow the same basic principals that we describe here. Class and method names may vary slightly from library to library. This document is meant as a general overview of the concepts shared across the different language bindings.

If you need code examples right away:

• Java
• NodeJS
• PHP
• Python
• Ruby
• .Net

Client Library Details (Beta)

We're gradually building our low-level documentation for each client library. This section is still in beta but you're very welcome to look:

• Java
• NodeJS
• PHP
• Python
• Ruby
• .Net

Basic objects

User

All interaction with the API libraries begins with an instance of the User class. Most other objects you will create while using the API require you to pass a User object, to which you have given your DataSift login and an API key.

All other objects that use the DataSift API require a User object for authentication. The User class provides methods for most ways you might want to create those other objects.

Definition

A Definition object represents a stream definition. These are roughly equivalent to the streams as you see them on the DataSift web interface, except that you cannot store them in your DataSift account for later retrieval.

Historic

A Historic object represents a Historics query. These are also roughly equivalent to the queries you can create via the DataSift web interface.

StreamConsumer

This object creates and manages connections to the DataSift streaming server. The default connection type used is HTTP, but some of the libraries also support WebSockets. This object is roughly equivalent to the code that runs in the browser when you're receiving data from a stream in the DataSift API.

When you create a StreamConsumer you supply a way to receive events from the stream. Events include connection, interactions, errors, status messages and more.

Exceptions

All of the libraries throw exceptions (or their language's equivalent) when errors occur. It's important that you catch and handle all possible errors for every call you make.

Putting it all together

This section is in pseudocode but you can translate it almost directly into your language of choice.

We start by creating a User object.

We now have a choice depending on what we actually want to do. We can create a StreamConsumer directly from the User object if we have a stream hash that we want to consume. For the sake of example we're going to say that we don't have the hash, we have some CSDL that we want to use. To do that we need to create a Definition object to represent that CSDL.

The first thing we should do is validate that the CSDL we've supplied is valid. We do this using the validate() method on the Definition object. As mentioned above, if the validate() method encounters an error (for example, if compiling the CSDL fails) it will throw an exception, so we make sure to catch that. However, we must also make sure we catch other errors that may occur. The example below uses a generic catch block to handle things like authentication or connectivity problems, but in your production code you should always catch specific exceptions.

Exception handling will be omitted from the remainder of this document, but please make sure you are correctly handling all exceptions that may be thrown by the library in your code, otherwise your program may terminate without warning while you're sleeping, and you'll miss out on some of the lovely data you want to consume.

Now that we know our CSDL compiles properly we can move on to creating a context for consuming data via a streaming connection. We start by creating our event handler.

For most libraries there is a reference class (or interface) which your event handler must implement. This defines the methods that must exist and the parameters they take. The most notable exception to this is the Ruby library which currently uses blocks rather than an event handler class.

The first method is onConnect() which gets called when a connection is successfully established with the DataSift streaming server.

The opposite of this event is getting disconnected. There's an event for that, too.

Let's take a moment here to look at the parameter being passed to these two events. All events will get the StreamConsumer object which is raising the event as the first parameter. This enables the handlers to make calls on the StreamConsumer.

The data coming down the streaming connection consists of a mixture of interaction objects, status messages, warnings and errors. We have events that handle each of these.

Status messages trigger the onStatus() method.

Status messages can contain additional information and this will be passed in the info HashMap. For example, a status of type "progress" for Historic queries will contain the percentage complete in info['progress'].

Errors and warnings trigger the following methods.

The data being received (interactions) trigger one of two methods: onInteraction and onDeleted. Essentially the data passed for a deletion notification is in the same format as normal interactions but only contains the data required for you to identify the interaction that has been deleted so you can delete it from your own storage systems.

Note that properly handling delete notification is required for you to remain compliant with some of the licenses you have signed.

Interactions trigger the onInteraction() method.

And that completes the EventHandler class.

Now that we have an event handler ready to receive events we can get a StreamConsumer from our Definition object.

The first parameter is the type of consumer we want. Most of the libraries only support HTTP streaming at this time, but some also support WebSockets. The second parameter is an instance of our EventHandler class.

We can now start to consume data. In most of the libraries this call will not return, so if your program needs to do other things while connected to the stream you'll need to wrap your usage of the API library in a thread.

The library will now compile the definition if necessary, connect to the streaming server, and start consuming data.

MultiStream support

The above discussion focused on consuming a single definition. Most of the libraries support consuming multiple definitions through the same stream connection. When doing that the event handler methods will get passed the hash of the stream which matched the interaction in addition to the other parameters.

Notes

• Make sure you have a DataSift login and an API key.
• The libraries will throw exceptions when something goes wrong, so make sure you're catching and properly handling them.
• For most libraries, once you start the StreamConsumer it will not return until the stream gets disconnected.
• You can control the StreamConsumer from any of the event handler methods using the StreamConsumer object they are passed.

Community-built libraries

We're happy to see that the DataSift development community has already started to add to the set of libraries we provided at launch.

Other code examples

• Saving to a database using PHP

On top of concentrating the raw data of multiple social media channels into a single stream, DataSift adds value to each data object it channels by augmenting it with other useful data. For example the standing of the author may be available, or the sentiment or language of the message. This enables you to write more incisive filters and receive more valuable results.

There are different types of augmentation:

• the expansion of any links included in the message and title of the pages to which they point
• results of sentiment or language analysis of the data object's message
• profile data of author or subject tracked by other providers such as Klout or Trends

DataSift uses data and analysis from the following providers to bring this extra insight:

​Not all data sources receive all types of augmentation. Language, Salience and Links are available for all sources; the other augmentations are currently availabe only for Twitter:

You access DataSift's APIs in two ways:

• Use HTTP headers
• Place all the parameters in a GET/POST request

HTTP Headers

This is the recommended appropach for production environments.

You need to include a header that contains: "Auth: username:apikey"

Most client libraries already support this authorization system.

GET/POST Requests

Accessing a DataSift API endpoint using GET/POST parameters is the simpler approach. It is sometimes useful for testing. Here's an example that calls the REST API to compile a CSDL filter:

    https://api.datasift.com/compile?csdl=<csdl>&username=<username>&api_key=<api_key>

Note that when you hit the Streaming API using the Websocket protocol, you must authenticate in the URI using GET parameters - there is no other option. In this case, the URI format is:

    ws://websocket.datasift.com/<hash>?username=<username>&api_key=<api_key>

Basics

What's an API?

The acronym "API" stands for "Application Programming Interface". An API is a defined way for a program to accomplish a task, usually by retrieving or modifying data. In DataSift's case, we provide API methods for all the core functionality. Programmers use the DataSift API to build applications that work with our platform. Programs talk to the API using HTTP or websockets.

How do I find my API key?

NOTE: your API key and DataSift username are both case sensitive.

To use the DataSift API, the first thing you need is your API key. 

1. Log in to DataSift.

2. Go to the Dashboard or to the Settings page.

3. Click the Copy to Clipboard icon under "Developer API Key".

Note that DataSift does not display the API key until you have purchased credits. If you have no credits yet, you cannot make an API call.

How do I find the hash for my stream?

1. Log in to DataSift.

2. Click on the Streams tab.

3. Select a stream.

4. Click the green "Use stream" button.

What's an interaction?

In DataSift, an interaction is one object from a source website. For example, an interaction from Twitter is a Tweet, including all the meta information such as the name of the author, the number of followers the author has, the number of people the author follows, and so on. It can also include augmentation information such as the language in which the content is written and the sentiment, positive or negative, conveyed in the message. An interaction is typically delivered to your client application as a JSON object.

What's a filter?

Filters sit at the very heart of DataSift's engine. You write them in our CSDL programming language. You can think of a filter as the logic that decides which input objects DataSift will deliver to you and which ones it will discard.

What's a stream?

A stream is the output of a filter. The terms are quite close. In fact, you might hear some developers refer to filters as streams.

What's a target?

A target is an individual field of information supplied by one of our social media partners sich as Twitter, by a third-party augmentation such as Klout, or by additional processing performed by DataSift itself, such as language analysis or gender detection. For example:

• twitter.text contains the 140 characters of a Tweet
• klout.score contains an author's overall score on Klout
• language.tag contains the 2-character language code that identifies the language in which the post of writter

What's a post?

A post is a generic DataSift term for a message from one of our social media partners. Posts can be Tweets, blog entires, blog comments, Myspace content, and so on.

What's CSDL?

CSDL is our programming language, the Curated Stream Definition Language. Every DataSift developer learns CSDL because it is the language you use to write filters. It's a very simple, compact compiled language that runs exceptionally quickly and is easy to learn.

How many streams can I create?

There is currently no limit to the number of streams you can create through our Streaming or REST APIs. You can create a maximum of 1,000 streams through our GUI. For full details on our API usage policy, please see API Rate Limiting.

What's JSON?

JSON (JavaScript Object Notation) is the default format for the data that our APIs return. Wikipedia introduces JSON as "a lightweight text-based open standard designed for human-readable data interchange."

How do I switch from JSON to JSONP?

To use JSONP you need to understand how to:

• Write a JavaScript callback function
• Include that function in your API call
• Make sure you specify JSONP in your API call

You define your JavaScript callback function on your web page and name it as your src parameter when you call a DataSift API. The callback function needs to be able to process standard DataSift objects but also be able to handle the other types of message that it might receive, such as ticks or error messages. A tick simply indicates that a connection is open but receiving no data. Ticks look like this:

    {"tick":1336057708,"status":"initialised","message":"Waiting for data"}

Your API call needs to include the name of the callback function. For example, suppose that you're using this API call with JSON and, to keep things simple, suppose that your username is "me" and your api_key is 888:

    http://stream.datasift.com/usage?username=me&api_key=888

To move from JSON to JSONP, with a callback function called xyz, you would change this API call to this:

    http://stream.datasift.com/usage.jsonp?username=me&api_key=888&callback=xyz

Notice that we changed the endpoint from usage to usage.jsonp to instruct DataSift to use JSONP.

I need something!

How do I save DataSift objects to a database?

We've created a PHP example to illustrate how to write code that works with a database.

Where are the API Client Libraries?

They're held on GitHub, the code-sharing site. Here's what they say about their site: "GitHub has grown into an application used by over a million people to store over two million code repositories, making GitHub the largest code host in the world."

Our Client Libraries page has the up-to-date list of the libraries we offer.

How do I keep up with changes to the API?

We make all our development announcements on Twitter. Just follow @DataSiftDev. We'd love you to follow @DataSift, too.

Something isn't working!

Is your service down?

We are constantly striving to build the best platform to unlock the power and data of social media. From time to time it may be necessary to disrupt our services to perform maintenance and upgrades. You can check on the status of DataSift, and check any scheduled work on the DataSift Status Dashboard. Any changes to the platform status are also announced on @DataSiftAPI

What do I do when I hit the rate limit?

The rate limit is designed to ensure that everyone plays fair. Essentially, you can use the Streaming API and the /stream endpoint of the REST API without hitting limits but for activities such as compiling or validating CSDL code through the REST API, DataSift applies limits. If you find yourself hitting the limits, you might have to wait for up to one hour. Each REST API endpoint has its own rate limit cost.

Why did the hash for my stream change?

If you're working in the GUI, it's because you edited the stream. If you're compiling using the /compile endpoint of the REST API, it's because you sent new CSDL. The bottom line here is that if you change the CSDL, the hash will change.

Basics

How do I determine the total cost of running a stream?

The total cost of running a stream depends on the complexity of the stream (which we measure in DPUs), how long you run the stream for, and the number of output objects it produces.

For example, each Tweet that we deliver to you costs $0.0001. If you create a stream that costs 1 DPU and run it for 6 hours, and receive 4,000 Tweets, the cost will be:

       1 * 6 * 0.20 + 0.0001 * 4000  =  $1.60

You cannot predict the license fee in advance because it is impossible to predict how many messages users will post. News stories, by their very nature, often come as complete surprises. However, you might decide to make an estimate of the traffic a stream will generate by running a test, either via DataSift's API or GUI, for a few minutes and extrapolating those results.

How do I determine the total cost of running a Historics query?

The cost of running a Historics query depends on data processing usage plus the licensing costs. Data processing usage is calculated based on the duration of the Historics query and the sample size of the output data; it is deducted from the monthly DPU usage. Licensing costs depend on the volume of data retrieved.

For example, you created a Historics Query of a simple stream of Tweets that costs 0.1 DPU, for the timeframe of one month, with 10 percent as the sample output data size. The data processing usage for this Historics query is calculated to be 288 DPU and it is deducted from your monthly DPU allowance. As per the usage statistics, the volume of data retrieved is a total of 1,212,194 augmentations and sources. Since Twitter charges $0.10 for every 1000 Tweets that we retrieve for the query, hence the licensing costs of the Historics Query will be $121.21 approximately. 

What are licensing fees?

DataSift charges licensing fees on behalf of our partner sites such as Twitter. The license fee that you pay is exactly proportional to the number of objects your stream produces. Or in case of Historics queries, the number of objects retrieved by your query.

If want to create very highly targeted streams, you should typically expect to receive a low volume of data and so your license fees will be very low. For example, a filter for Tweets about hippopotamuses sent by authors with an unusually high Klout score within a radius of 20 miles of San Diego zoo, isn't going to generate very much output, even on days when the hippos do something exceptional.

On the other hand, a filter that looks for any mention of, say, music will probably generate substantially more output and cost more in license fees.

What's a DPU?

A DPU is a Data Processing Unit, a reflection of the computational complexity for the processing that you perform on the DataSift platform. A higher number represents a more complex stream. We measure DPUs on a per-hour basis because running a stream for five hours costs five times as much as running it for one hour.

What is the cost of running a 1 DPU stream for one hour?

It costs 20 US cents. If you purchase a subscription, you will benefit from a discount.

How long does it take to run a Historics query?

It depends on the capacity of our data archive at the time of creating a Historics query, as well as the timeframe and sample size of your Historics query. The timeframe of the query is the duration between the start date and time, and the end date and time of the query. The sample size of the output data can be either 100 percent or 10 percent of all the available data.

When you create a Historics query, it needs to access our data archive in order to retrieve output data for a selected timeframe. Our data archive could be very busy when multiple Historics queries are running at the same time. If the data archive is running over capacity, your query will be queued; that is, it will have to wait for access until other queries accessing the data archive have been executed. Although the queuing process takes a little time, keep in mind that Historics queries, once they are running, retrieve data 30 times faster than a real-time filter.

Once your Historics query has access to our data archive, it then depends on the timeframe of your query and the sample size of the output data. A Historics query with a shorter timeframe and a sample size of 10 percent is likely to execute more quickly than a query with a longer timeframe and a sample size of 100 percent.

What if I stop a Historics query midway? Will I still be charged for it?

Yes, you will be charged even if you stop a Historics query midway. You will be charged for the licensing costs of the volume of data retrieved until you stopped the query. The data processing usage until you stopped the query will also be deducted from your monthly DPU usage.

What's the difference between credits and DPUs?

You purchase credits on the DataSift platform. Credits are priced in US dollars.

A credit costs $1. One credit is equivalent to 5 DPUs so the effective price of one DPU is 20 cents.

Why do you charge using DPUs?

DataSift believes customers should only pay for what they consume. DataSift is a cloud platform, allowing you to consume only what you need and retain the flexibility to scale, either up or down, whenever necessary. The DPU amount that you pay is determined by the complexity of the rules you create.

How can I optimize my streams to use fewer DPUs?

Visit our Optimization page first. If you need further guidance, please contact our Support team.

How can I check my billing?

Via the REST API:
     For Streams, hit the /dpu endpoint to find details of DPUs for a stream.
     For Streams, hit the /usage endpoint to find how many objects DataSift has delivered to you.
     For Historics queries, hit the historics/prepare endpoint to get details of DPUs for your Historics query.   

Via the DataSift GUI, visit our Billing page.

Why is my DPU cost higher than I expected?

The minumum DPU charge is currently 1 DPU per hour, no matter how simple your stream is.

If you run just one stream that costs just 0.1 DPU, the total charge is 1 DPU per hour, which equates to 20 cents. In other words, the minimum DPU cost to use the platform is 20 cents per hour.

However, if you run ten streams, and they all cost 0.1 DPU, DataSift will still charge only 1 DPU per hour for all ten.

Why can't I pay a fixed price?

You can pay a fixed price for the processing but not for the licensing. We offer a range of subscriptions which include prepaid DPUs. The license cost of the content is variable and depends on the number of objects your stream returns. Clients who choose to prepay for DPUs benefit from a discount.

What is the charge for a Historics Preview?

A Historics Preview is charged at a fixed cost of 20 DPUs per request. 20 DPUs are deducted from your account after a complete and successful execution of your request. 

Terms of Service

How much latency does DataSift introduce?

Applications succeed or fail based on performance. DataSift dynamically distributes workload into under-utilized CPU and memory resources to provide best-in-class service delivery with very low latency. DataSift consistently performs faster than competitors. A new Tweet, for example, is likely to be available on our platform 1 to 2 seconds after it appears on Twitter.

Do you offer an SLA?

Yes! We are able to offer the most reliable and comprehensive SLA to our customers because we host our own dedicated cloud infrastructure. It is built on a massive-scale Service Orientated Architecture giving our customers peace of mind that we will continue running when others fail.
For more details, please take a look at our Terms and Conditions.

Billing details are explained in full on our Terms page. The cost of a stream depends on how many operators you include. Some operators are more expensive than others. All streams have a fixed cost; some have a variable cost too because some data suppliers charge for their content. Billing for Historics also works in the same way. The following page will provide you with detailed information on how our billing system works. 

Don't forget to take a look at our Billing FAQ too.

Overview

You can compile and preview streams free of charge through the website GUI.

There is a charge to use streams through our APIs. The cost of using a stream via the API is a function of two variables:

Payment plans

There are two types of payment plan which differ in the approach to charging for the data processing cost, allowing you to optimize according to your usage pattern:

Whereas a rule's data processing rate is certain as soon as it is defined, its throughput is impossible to predict, it can only be estimated. You might want to run some sample executions to get a feel for the throughput cost of a stream.

Notifications

The DataSift billing system calculates the cost of using streams from the DPU rate and licensing costs. DataSift also allows you to monitor your usage by enabling notifications via email and the Dashboard. The notifications vary depending on the type of payment plan you are on.

On Demand

If you choose the On Demand plan, you will receive notifications if your credit balance runs low or falls to zero.

Monthly subscription

In a monthly subscription, you can set a variable cost limit on your account. You will receive notifications when you are close to and if you reach your variable cost limit. You can set or change your variable cost limit any time during the billing cycle.

The first notification is triggered when you have used up 80 percent of your variable cost limit. For example, if you set your variable cost limit to $2,500, you will receive the first notification when you have used up $2,000 on your account. You will receive the second notification when you reach your variable cost limit, at which point we will stop your streams. It is good practice to monitor your usage and ensure that your variable cost limit is always high enough for you to be certain that you will not have any problems for the duration of the month.
 

                                                                             Preview of notifications in Dashboard

                                                                             Preview of notifications via email
 

If you notice that you are close to your variable cost limit and then you raise it, you might be below 80 percent of the new limit or you might be above 80 percent of the new limit; it all depends on where you set your new limit.

For example, if you set the variable cost limit to $2,000 on your account, you receive the first notification when you have used up $1,600. Suppose that you receive that notification and you raise the variable cost limit to $2,500. There are two possible scenarios to consider:

• If you are below 80 percent of the new variable cost limit, which is $2,000, you would receive both the notifications: a warning when you reach 80 percent of the new variable cost limit and then a notification when you reach your variable cost limit.
• If you were above 80 percent usage, you will only receive a notification when you reach your new variable cost limit.

Billing for Historics queries

You can use Historics queries if you are on a monthly subscription, subject to one-time activation by your account manager. The cost of running a Historics query depends on data processing usage plus licensing costs, and the original DPU complexity of the stream you are running the query on.

Data processing usage for Historics is calculated based on the duration and sample size of the output data. The duration of the query is determined using the timeframe of the query, that is the duration between the start date and time, and the end date and time of the query. The sample size of the output data can be either 100 percent or 10 percent. For all Historics queries, there is a premium on the DPU usage compared to usage for live streaming. DPU usage for the 100% sample size is 125% of what you would pay for live streaming of the same filter. Similarly, for the 10% sample size, the DPU usage is 40% of what you would pay for live streaming of the same filter.

Hence, when you create a Historics query, DataSift is able to calculate the DPU usage before the query is executed. This DPU usage information is displayed on the Confirm New Historic Query page. When running a Historics query through the Historics API, you need to hit the historics/prepare endpoint to create a Historics query and get the total DPU breakdown for your Historics query before it is executed. DPU usage charges are deducted from the monthly DPU allowance. 

On the other hand, licensing costs are calculated based on the volume of data retrieved for a particular Historics query. For a given CSDL filter, licensing costs for a Historics query of 100 percent sample size will be more than for a Historics query of 10 percent sample size.

You can view usage statistics for Historics queries on the Billing page. You can view total licensing costs and the DPU usage for your Historics queries. You can also view the volume of data retrieved by a Historics query and the number of Historics hours used. Alternatively, you can hit the usage endpoint in DataSift API which will give you a more accurate figure for the number of objects processed.
 

          

Billing for Historics Preview

Historics Preview is available for all accounts, on any payment plan, be it Subscription or Pay As You Go. Each request has a fixed cost of 20 DPUs. There are no licensing fees charged for a Historics Preview since you will not be actually receiving any interactions matching your filter. You will ony receive aggregate statistics for your selected filter.

The 20 DPUs are deducted from your account only after a complete and successful execution of your Historics Preview request. If your request gets interrupted while it is being processed, you won't get charged. You can only request a single Historics Preview per stream; if you request a new one, the previous request is overwritten.
 

Billing information

Find your DPU cost via the GUI

In DataSift's GUI you can check the DPU breakdown:

1. Select a stream

2. Click View Definition

The DPU breakdown appears below your CSDL code.

Find your DPU cost via the API

DataSift's REST API provides a dpu endpoint that gives the total DPU cost for a rule and the breakdown of its individual elements. 

    api.datasift.com/dpu

For Historics, DataSift's REST API provides a historics/prepare endpoint that gives the total DPU breakdown for a Historics query.

    api.datasift.com/historics/prepare

Cost of operators

Some operators in CSDL have a fixed DPU cost while others have a variable cost.

For fixed-cost operators you simply multiply the number of times you use the operator in a stream by its DPU cost. For example, if you use the contains operator twice in a stream the cost is 0,2 DPUs. 

Reg​ular Expressions 

The DPU cost of a regular expression is calculated as:

          cost = the number of characters in the expression divided by 100.

The minimum charge for one regular expression is 0.1 so, for example, a regular expression that includes 10 characters costs 0.1 DPUs while a regular expression that includes 100 characters costs 1.0 DPUs.

geo_polygon 

The DPU cost of a geo_polygon depends on the number of vertices it has. To determine the DPU cost of any geo_polygon, divide the number of vertices by 30.

For example, a hexagon has 6 vertices so it has a DPU cost of 0.2. A triangle has 3 vertices so it has a DPU cost of 0.1.

contains

The DPU cost for the contains operator is based on the number of values you match against and the way you use the operator.

Using the contains operator to find a phrase

    twitter.text contains "My dog ate my homework"

In this case, you can match against up to seven values for a cost of 0.1 DPU. The cost increases by 0.1 DPU as you add more words to the matching phrase. Here are the first few DPU cost bands.

For example this filter has just one word in the argument so it costs 0.1 DPU:

    twitter.text contains "iPad"

This filter has eight words in the argument so it costs 0.2 DPU:

    twitter.text contains "iPad is my favorite tablet device right now"

Using the contains operator to find individual words

in/contains_any

The DPU cost for the in and contains_any operators is based on the number of values you match against. The following table shows the DPU cost for any filter that uses these operators.

For example, this filter matches against 10 values so it costs 0.2 DPUs.

    twitter.text contains_any "apple, microsoft, hp, dell, oracle, google, yahoo, ebay, amazon, facebook"
 

The exact cost is determined using a sliding scale, so if you have 99 values in the command, the cost will be slightly lower than 1 DPU. Note that the table shows how we calculate DPU costs for a list of single keywords. In practise, you will often write filters that use the contains_any keyword with a list of phrases of varying length. For example:

    twitter.text contains_any "Yesterday, Yellow Submarine, The Long and Winding Road"

Since phrases take longer for DataSift to process than single keywords, the DPU cost is slightly higher. For example, a list of 30 single keywords with the contains_any operator incurs a DPU cost of 0.4. However, if you filter for 10 phrases, each of three words, the DPU cost is 0.5.

We recommend that you check the DPU cost before you run a filter. The /compile endpoint returns a JSON object that includes the DPU cost.

Tags

Operators used inside a tag statement are normally charged at 10% of their usual DPU cost.  

For example, if the normal cost of a rule is 1 DPU, that same code inside a tag statement would cost 0.1 DPU.

If the normal cost is less than 1 DPU, there is no charge.

For each data source, discover how to fine tune the data you receive.

DailyMotion and Youtube belong to the Videos family of feeds. Amazon, Flickr, IMDb, Reddit, Topix, and 2channel belong to the Boards family.

NoteThis section describes the data that you can consume from DataSift. It does not describe the targets that you can filter against. Read Targets or Output Data? to learn more.

How does it work?

Objects flow into DataSift one by one. Each of them contains a rich variety of data, much more than you might expect. Think about a Tweet for a moment. As well as the 140-character payload that most people would say 'is' the Tweet, the object contains the author's screen name and a link to their photograph, the number of followers they have, the number of people they follow, and more than 40 other nuggets of information. Each Tweet that you send contains your entire Twitter bio hidden inside, for example.

In DataSift's filtering engine, these individual fields are called targets. For example, if you write this filter:

    twitter.text contains "hippopotamus"

you're instructing DataSift to deliver every Tweet that contains the word hippopotomus and to exclude everything that isn't a Tweet and every Tweet that doesn't mention a hippo.

The targets and operators documentation describes filtering in detail, with examples of how to use every CSDL operator and every target. 

What data is in the output stream?

To use DataSift effectively, developers also need to understand the data that is actually delivered in the output stream and to know how to control what is delievered. This process is entirely separate from filtering.

Here are some key things to learn first. Don't worry, it's simple.

Programming APIs

We offer client libraries for the most popular programming languages.

Click to find out how to download a library, check which features it supports, and see step-by-step examples of how to benefit from our code.

• Java
• Node.JS
• PHP
• Python
• Ruby
• .Net

Our engineering team is always looking at new candidate languages. If your favorite language does not yet have a DataSift client library, contact Support for the latest news.

Community-built libraries

We're happy to see that the DataSift development community has already started to add to the set of libraries we provided at launch.

Other code examples

• Saving to a database using PHP

A key point to understand is that targets are not the same as output data:

• A target is something that you can filter on
• Output data is something that you receive from DataSift

You will encounter three different cases: when data is available in both a target and in the output, when it is available only as a target, and when it is available only in the output.

What fields appear in output streams?

Take a look at our complete list of data elements in your output streams.

Both

The most common case is when information is available both as a target and in the output. For example, when you filter for people on Twitter who have more than 1,000,000 followers, the output that you receive will include their exact follower count.

    twitter.user.followers_count > 1000000

In this CSDL code, twitter.user.followers_count is the target. 

You can see the output element, in JSON format, in the following code snippet:

In this example, your CSDL filter tells you nothing about the content that you will receive. The filter simply finds all Tweets posted by people with a large number of followers.

Target only

You will see cases where information is available as a target so you can filter on it, but it is not in the output. For example:

    interaction.sample < 10

Output data only

Finally, you will find cases where information is not available as a target, so it is not filterable, but it does appear in the output. For example: the created_at field is often presented in the output data but it is not a target.

If you enable the Salience augmentation, you will find that the Salience Entites data is delivered but you cannot filter against it.

DataSift is very simple to use. You simply build filters to find the information you want.

If you're new to the product, we suggest you read these pages first.

1. The filters are written in our programming language. It's called the Curated Stream Definition Language (CSDL).

2. The key feature of CSDL is its set of operators.

3. The filters take information from social media sites and augmentations. Together these are called targets.

4. You can write applications that access DataSift through our APIs.

5. You can test your filters in our user interface before you start to explore the APIs.

6. Please read our Understanding Billing guide and Billing FAQ.

7. Please read Targets vs Output Data to understand the difference between filtering data and consuming data.