The Query Builder

Jacek Artymiak | 24th January 2013

At DataSift we love open source. As part of our commitment, we're proud to announce that a major new component of the DataSift platform, the Query Builder, is now available. It's open source and you can download it from GitHub today. Take a look at our demo page to try out the Query Builder.

What is the Query Builder?

Everyone talks about Big Data, but not many people know how to handle it. We live it. We created the Query Builder to bring the advanced functionality of DataSift to business users.

We consume over a billion items per day, processing them, augmenting them with analytical data, and making them available in JSON format. The Query Builder includes a built-in dictionary that shows all 450 of the different targets that users can include in their DataSift filters, so even novices can get started right away.

The Query Builder is a code generator that produces SQL-like commands that users can share. It does everything via a point-and-click interface where users create queries visually. They can use the features of the Advanced Logic Editor, shown above, to build complex filters by combining simpler ones.

Responsive design and standards compliance for the post-PC era

You worked hard on your site and the last thing you want to put on it is an ugly widget that clashes with the rest of the page. Rest assured that we've put a lot of time and effort into making sure the Query Builder is standards-compliant, responsive, and ready for post-PC touch screen devices. We strive to follow the latest standards for good design, responsiveness, and programming, be they official or commonly agreed upon. In a browser, the Query Builder supports the latest versions of Internet Explorer, Firefox, Safari, Opera, and Chrome.

The Query Builder is built using standard tools and technologies (JavaScript, HTML5, JQuery and CSS). The responsive design fits a broad range of screen sizes; it's fully compatible with the iPhone, Android, and iPad as well as laptops and desktops. It even includes graphical assets for Retina-resolution displays. And since it works equally well with a mouse or a touch screen, filtering for answers in an ocean of a billion interactions is as easy as sending a Tweet from your iPhone.

Getting Started

The Query Builder project is hosted on GitHub. When you want to embed it on a web page, log into your server, change the working directory to the document root directory, and then clone the repository with a single command:

git clone

Alternatively, download the project archive and unpack it to the document root directory on your web server.

In both cases, you should end up with a directory that contains a number of subdirectories. Most of the time you will only need datasift-editor/minified, unless you want to do some deep modifications of the code base and the resources. But make sure you read our configuration guides before you do that; in most cases, you only need to make small modifications to the Query Builder object initialization code. This is done by overriding the exposed configuration options.

<script type="text/javascript" src=""></script>
<script type="text/javascript" src="jcsdl/jcsdl.min.js"></script>
<script type="text/javascript" src="query/jcsdl.definition.js"></script>
<link rel="stylesheet" type="text/css" href="jcsdl/jcsdl.min.css" />

<div id="jcsdl"></div>
<script type="text/javascript">
  var editor = new JCSDLGui('#jcsdl');

The Query Builder produces code just like a programmer would using a code editor, but in a user-friendly way. The code is based on DataSift's Curated Stream Definition Language, CSDL, with added machine-readable comments that allow it to work with the Query Builder. This enhanced version of CSDL is known as JavaScript CSDL, or JCSDL.

This process enables users to generate and share CSDL code without knowing how to program. All that power is available without having to learn how to write a single line of code. Simply clone the Query Builder repository or upload the files that the Query Builder needs to run to your server and add eight lines of HTML code to the page where you want to embed it.

Modular and highly-customizable by nature, the Query Builder is easy to embed on a web page, blog, or inside a web view in a desktop or mobile application. You can customize it to match a variety of requirements for integration and branding.

Customizing the Query Builder

The Query Builder code you can download today from GitHub is exactly the same code we use on our website. We give you full freedom of choice when it comes to the use of our code and the approach to implementation.

The simplest form of customization you would perform might be to make the Query Builder follow the look and feel of your site. This is easily done by overriding the CSS style definitions with your own modifications. If you want to go one stage farther, you can replace the Query Builder's graphical assets with your own. The design of the CSS stylesheet is optimized to facilitate quick changes with minimal effort. When you want to add your own CSS, simply import it after the original stylesheet and all will be well.

Next, you might decide to customize the functionality and behavior of the Query Builder. You can modify the responsiveness of the interface or narrow down the choice of the data sources available to users. Changes like these do not require extensive knowledge of programming and can be implemented quickly by someone with a little knowledge of JavaScript. You can find the example of reducing functionality of the Query Builder and a working demo on our developer documentation site.

We have added built-in help in the form of tool tips so that end users of the Query Builder can learn more about DataSift's targets and operators. These are downloaded directly from our servers, so any changes will appear on your users' screens as soon as they are published, without you having to do anything unless you want to jump in and create your own tool tips.

We also support you, the developer. We have a whole site dedicated to the subject of embedding, styling, and configuring the Query Builder.

Connecting to DataSift

Once your implementation of the Query Builder is fully operational, it's time to connect it to our plaform. You need to capture the JCSDL generated by your users, pass it on to DataSift, capture the results, and present them back to the user. You have full freedom to implement your own solution here as well as full freedom of user management.

This is where you can add a lot of your own creativity and value. Processing and presentation of the results is one important area where you can create your own tools and make your users happy. We have prepared a sample implementation to get you started. Read through the code, try it, see what it does, and create your own magic. And you do not have to worry about backward compatibility. If you follow our configuration procedures, upgrading your installation of the Query Builder will be as simple as unpacking an archive.

You are also free to manage your own users in any way you like. You can choose to require your users to provide their own DataSift credentials or you can use a single set of DataSift credentials for company-wide access without having to manage multilpe accounts. Or you could manage your users' accounts for them based on their internal credentials.

So, there you have it. Now go make something amazing and let the world know about it.

Previous post: Open Source Software at DataSift

Next post: Split-Second Social Media Analysis with DataSift and Redis