Security Best Practices


A successful application built on DataSift is likely to attract attention. Most of that attention will be good: users singing your praises, and developers complimenting your programming. Some of that attention, though, might be negative.

This page presents a set of ideas designed to put you on a path towards better security in your application. It's not the final word; far from it. If there's anything you'd like to see added to it, please let us know. If you've discovered a security issue that directly affects DataSift, please email



The following threats are applicable no matter what your platform.

Password Retention

Authentication in DataSift requires the API key and username.

If you are retaining API keys, consider encrypting them. One option is bcrypt-ruby, but there are lots of other ways to store encrypted information.

Input Validation

Don't assume that your users will provide you with valid, trustworthy data. Sanitize all data, checking for sane string lengths, valid file types, and so on. DataSift attempts to sanitize data POSTed to our APIs, but a little client-side help goes a long way. Whitelist the types of input that are acceptable to your application and discard everything that isn't on the whitelist.

Exposed Debugging Information

Be sure that you're not exposing sensitive information through debugging screens/logs. Some web frameworks make it easy to access debugging information if your application is not properly configured. For desktop and mobile developers, it's easy to accidentally ship a build with debugging flags or symbols enabled. Build checks for these configurations into your deployment/build process.

Inadequate Testing

Ensure that your tests (you do have tests, right?) check not just that you can do what should be able to do, but that bad guys can't do what they shouldn't be able to do. Put yourself in an attacker's mindset and whip up some evil tests.

Not Letting People Help

Have you set up Do those emails go right to your phone? Make it easy for people to contact you about potential security issues with your application. If someone does report a security flaw to you, be nice to them; they've just done you a huge favor. Thank them for their time and fix the issue promptly. It's fairly common for security researchers to write about vulnerabilities they've discovered once the hole has been closed, so don't be upset if your application ends up in a blog post or research paper. Security is hard, and nobody is perfect. As long as you're fixing the issues that are reported to you, you're doing right.

Consider hiring security professionals to do an audit and/or penetration test. You can't depend solely on the kindness of strangers; for every vulnerability that someone was nice enough to report to you, there's ten more that malicious hackers have found. A good security firm will dig deep to uncover issues. Look for firms and individual consultants that do more than run a few automated tools.

The Law

If your application is (going to be) handling money, you may be required by law to adhere to certain security practices and regulations. Find out what's applicable to you and make sure you're up to code.

Web Application Security

Unfiltered Input, Unescaped Output

One easy-to-remember approach to input validation is FIEO: Filter Input, Escape Output. Filter anything from outside your application, including DataSift API data, cookie data, user-supplied form input, URL parameters, data from databases, etc. Escape all output being sent by your application, including SQL sent to your database server, HTML to you send to users' browsers, JSON/XML output sent to other systems, and commands sent to shell programs.

Cross-Site Scripting (XSS)

XSS attacks are, by most measures, the most common form of security problem on the web. In short, if an attacker can get their own JavaScript code into your application, they can do bad stuff. Anywhere you store and display untrusted, user-supplied data needs to be checked, sanitized, and HTML escaped. Getting this right is hard, because hackers have many different ways to land XSS attacks. Your language or web development framework probably has a popular, well-tested mechanism for defending against cross-site scripting; please make use of it.

Generally: if HTML isn't needed from some user-facing form, filter it out; for example, there's no reason to allow anything other than integers when storing a phone number. If HTML is needed, use a known-good whitelist filter. HTMLPurifier for PHP is one such solution. Different contexts may require different filtering approaches. See the OWASP XSS Prevention Cheat Sheet for more on filtering.

SQL Injection

If your application makes use of a database, you need to be aware of SQL injection. Again, anywhere you accept input is a potential target for an attacker to break out of their input field and into your database. Use database libraries that protect against SQL injection in a systematic way. If you break out of that approach and write custom SQL, write aggressive tests to be sure you aren't exposing yourself to this form of attack.

The two main approaches to defending against SQL injection are escaping before constructing your SQL statement and using parameterized input to create statements. The latter is recommended, as it's less prone to programmer error.

Cross-Site Request Forgery (CSRF)

Are you sure that requests to your application are coming from your application? CSRF attacks exploit this lack of knowledge by forcing logged-in users of your site to silently open URLs that perform actions. In the case of a DataSift app, this could mean that attackers are using your app to force users to post unwanted messages or follow spam accounts. You can learn more about this sort of attack on PHP security expert Chris Shiflett's blog.

The most thorough way to deal with CSRF is to include a random token in every form that's stored someplace trusted; if a form doesn't have the right token, throw an error. Modern web frameworks have systematic ways of handling this, and might even be doing it by default if you're lucky. A simple preventative step (but by no means the only step you should take) is to make any actions that create, modify, or destroy data require a POST request.

Lack of Information about Threats

If you think there's an issue with your web application, how do you find out for sure? Have critical exceptions and errors emailed to you and keep good logs. You may want to put together a dashboard of critical statistics so that you can see at a glance if something is going wrong (or staying right).

Desktop Application Security

We could use suggestions from desktop developers about the security issues they've run into. Developers working in sufficiently high-level languages shouldn't be dealing with buffer overflows and the usual security issues. What have you defended against?

Unencrypted Storage of Credentials

Once you have an API key for a user, where do you keep it? Ideally, in an encrypted store managed by your operating system. On Mac OS X, this would be the Keychain. In the GNOME desktop environment, there's the Keyring. In the KDE desktop environment, there's KWallet. In Windows, use a two-way encryption tool.

Security Resources

The below links are great ways to learn more about security. Security is a deep topic, but don't feel like you have to learn everything about it before taking proactive steps to lock down your application. A little security-mindedness goes a long way.


  • Purdue CERIAS - Center for Education and Research in Information Assurance and Security
  • US-CERT - current identified threats for various platforms and applications

Web Application Security

Desktop Application Security