Platform Performance Gains with Arista Switches
In late 2012 I wrote about the migration of DataSift's Hadoop cluster to Arista switches but what I didn't mention was that we also moved our real-time systems over to Arista too.
Within the LAN
During our fact-finding trek through the Cisco portfolio we acquired a bunch of 4948 and 3750 switches which were re-purposed into the live platform. Unfortunately, the live platform (as opposed to Hadoop-sourced historical data) would occasionally experience performance issues due to...
Read Platform Performance Gains with Arista Switches
New Release of the Query Builder
Have you tried our Query Builder yet? It's a visual tool that makes it easy for newcomers to get started with DataSift quickly, before they even begin to learn our query language, CSDL. Despite its simplicity, the Query Builder offers very nearly all the features on offer in the full language. It includes every CSDL operator and logical operator, together with very nearly all the targets and augmentations. Recently we added the ability to use...
Read New Release of the Query Builder
Historics - We Keep Making It Better
A year ago, Datasift released Historics, a product that enabled users to access content from the past. Its demand has grown massively over the past year. We have had to make many optimizations in order to keep up with not just the demand, but the scale of our ever-growing archive.
Our Historics archive is very close to one petabyte in size now and we are adding about two terabytes to it each day. We run over 2,000 Hadoop jobs every month that scan over a total of nine trillion records...
Read Historics - We Keep Making It Better
Writing CSDL in Vim
Like any developer-friendly company, DataSift too has fans of the good old Vim editor working for us and with us. And since we spend so much time inside Vim, it is no wonder that we use it to write CSDL too. Which is why today I'm especially happy to announce that CSDL syntax highlighiting has been added to the Vim source code repository and should be shipping with all major operating systems worth using soon. (An OS worth using is the one that ships with Vim, of course.)
Read Writing CSDL in Vim
Split-Second Social Media Analysis with DataSift and Redis
Social media gives us a way to sample trends and sentiment in real time. Consequently, it is very important that the analysis of the data we are looking at also happens in real time. And we want to help you, because here at DataSift we want our platform to be the Swiss Army knife of the social media analysis tools. We try to be flexible and do as much of the hard work as possible so that you can focus on analyzing the data instead of having to think how to feed it into your processing pipeline...
Read Split-Second Social Media Analysis with DataSift and Redis