Multiple streams

Updated on Friday, 5 April, 2013 - 14:21

The streaming API allows you to subscribe to multiple streams simultaneously using either HTTP streaming or websockets. The endpoints to access multiple streams are:

  • ws://websocket.datasift.com/multi
  • https://stream.datasift.com/multi

We provide client libraries for:

  • Java for websockets streaming or HTTP streaming
  • Node.JS or PHP for HTTP streaming only

These are the only languages that we currently support for multi-streaming.

You can run up to 5,000 streams simultanously over a single connection.

Opening a connection

There are two ways to open the initial connection; you can open it without specifying any streams, or you can open it with some streams.

 

Here's an example that opens an HTTP connection without any streams:

    https://stream.datasift.com/multi

 

This one opens a connection with three streams and happens to use the Websockets protocol:

    ws://websocket.datasift.com/multi?hashes=hash1,hash2,hash3

 

Take a look at our API Authentication page for more information.

 

Using messages to subscribe/unsubscribe streams

When a connection is open, you can send "subscribe" messages to add new streams or you can send "unsubscribe" messages to stop receiving data from them. You can subscribe or unsubscribe any time you want.

Remember to use the correct protocol and endpoint depending on which DataSift client library you're using:

  • https://stream.datasift.com/multi using the Node.JS client library
  • ws://websocket.datasift.com/multi using the Java client library

 

Note that the PHP client library does not support subscribe/unsubscribe messages but you can specify multiple streams when you open the connection like this:

    https://stream.datasift.com/multi?hashes=hash1,hash2,hash3

 

The messages are simply JSON objects containing subscribe or unsubscribe messages with the associated hash.

Example subscribe message:

    { "action" : "subscribe" , "hash": "4f3af38ad1c31646561bc814c3e69af5"}

Example unsubscribe message:

    { "action" : "unsubscribe" , "hash": "4f3af38ad1c31646561bc814c3e69af"}

Send one message for each stream you want to open or close.

 

Return Objects

The response body contains a list of JSON objects separated by new lines.

Interactions

When an interaction has matched your CSDL it will be sent as an interaction JSON object encapsulated with the associated hash.

    {"hash":"123456abcdefg", "data": {"interaction":{...}}}

 

Errors and Warnings

If an error occurs, DataSift will end the connection.

For instance, a 401 Unauthorized Error returns a JSON object describing the problem:

    { "status": "error", "message": "An error message description will be here" }

Occasionally a status message might be sent in response to a request to the service. For instance, if you have too many subscriptions in one connection you will receive a warning like below. Warning messages will not close the connection.

    { "status": "warning", "message": "A warning message description will be here" }

 

Consult the complete list of error messages.