Blog

New delivered_at meta field for Push

DataSift is adding a new metadata field to each JSON object delivered via Push in the json_meta output format - a delivered_at timestamp. This new timestamp represents the time DataSift delivered this particular object. An example of a json_meta formatted Push delivery containing this new field can…

Read New delivered_at meta field for Push >

Bitly, Links, and Metadata

Today, Datasift announces two new link resolution services. First, we're delighted to be partnering with bitly, the #1 link sharing platform, which powers 75 percent of the world’s largest media companies and half of the Fortune 500 companies. With over 20,000 white-labeled domains, bitly generates 2…

Read Bitly, Links, and Metadata >

Open Graph and Twitter Cards

Today we announce our new data source, the bitly input stream. With 200M clicks a day, it provides an excellent augmentation to the links embedded in Tweets and in messages from other data sources. In the past, we could see which content was being shared; we can now see which links are actually…

Read Open Graph and Twitter Cards >

Historical Architecture - Data Mining Billions of Tweets

The technology behind the platform In addition to the existing live and buffered streaming, DataSift can now record data to its massive storage cluster. The core of the recording platform is an HBase cluster of more than 30 nodes, with over 400TB of storage. Every piece of information is replicated…

Read Historical Architecture - Data Mining Billions of Tweets >