CSDL comes with a range of advanced features that allow much more complex filtering through the ability to include parent streams and also use our data processing to perform 'tagging' of content to remove the burden of doing it on the client side.

There's only one limit and most developers are unlikely to run into it: the maximun length of the CSDL code in your filter must not exceed 1 MB. Fortunately, there's an easy workaround that employs the stream keyword. If your stream reaches the point where it exceeds 1 MB, you can easily have it call one or more other streams. In this way, you can distribute your code across many chunks of CSDL code. 

Here are some additional keywords and techniques you can employ to optimize streams, reduce costs, and create even more complex and powerful filters.

	Tag keyword   Use the tag keyword to add metadata to the output objects coming from a stream. Advanced CSDL developers employ the tag keyword very frequently because it makes subsequent analysis easy.
	Stream keyword Use the stream keyword to include an existing stream in a new stream definition or merge two or more streams.
	Regular expressions Use regular expressions to create super-powerful stream definitions.
	Selecting data sources   You can select or exclude individual social media sites in a variety of ways.
	Optimization You can optimize your CSDL code to run more efficiently and to minimize costs.
	Sampling data You can sample data rather than drinking from the entire firehose. For instance, you can create a filter that samples just a percentage of the input objects flowing into DataSift. This approach is particularly useful if you're performing statistical analysis where, for example, just 10 percent of the data might be enough to form a representative sample.

 

You can add comments to your CSDL code using a style similar to the syntax used in C.

Comment out entire lines like this:

Comment out part of a line like this:

Or add a mid-line comment like this: 

Note: in most cases a stream's hash changes whenever you edit the CSDL for that stream. However, adding comments to CSDL does not cause the hash to change because comments are normalized out when the stream is compiled.

CSDL accepts comments anywhere it accepts whitespace. Don't insert comments within quoted strings as the CSDL parser disables white space skipping here

Whitespace in the CSDL compiler

Our definition of whitespace is derived from the isspace() function in C. We use:

• space
• tab
• newline
• feed
• carriage return

Word Matching using Contains/Any

DataSift tokenizes each interaction target. It ignores white space but treats every punctuation symbol (as defined in ispunct()) as a separate word.

For example, if the input is:

    This is,    a test    

The output is:

    <This> <is> <,> <a> <test>

Using this technique we can match words without allowing punctuation to affect the boundry of what is considered a word, yet still allows you to include punctuation in filter when you want to, as the punctuation is not stripped from the text.

This page is for advanced DataSift users. The filtering engine that runs your CSDL code is tuned for efficiency in order to handle massive volumes of data. To accomplish this, it performs some preliminary steps, temporarily stripping all @mentions and URLs from posts and then tokenizing the remaining content into "chunks" before it performs any filtering.

Let's look at these processes in turn.

@Mentions

DataSift's filtering engine ignores all mentions of Twitter users in Tweets. Therefore, it would see this Tweet:

    If you like @DataSift you have to follow @DataSiftNews and @DataSiftDev every day.

as this:

    If you like you have to follow and every day.

Of course, this means that you cannot write a CSDL filter like this to look for mentions of @DataSift:

 

Instead, use the twitter.mentions target like this:

 

 

URLs

URLs are also invisible to the filtering engine. DataSift sees this Tweet:

    If you like @News surf to http://nytimes.com.

as this:

    If you like surf to.

Consequently, you cannot write a CSDL filter like this one for URLs:

 

Instead, you need to use a Links Augmentation target like this:

 

 

Cashtags

DataSift uses 'cashtags' to allow you to filter for stock ticker symbols; for example:

This works because we have chosen to treat the $ symbol is treated as an alphabetical symbol, not punctuation.

 

Chunking

The engine also separates the characters it sees into "chunks". Characters can be:

• alphanumeric
• whitespace
• punctuation

Let's look at alphanumeric and whitespace characters first. Every time DataSift finds one or more whitespace characters between alphanumeric strings, it starts a new chunk. For example, it would split this string into three chunks:

    The Polar Express

The chunks are:

• The
• Polar
• Express

It also treats punctuation as a chunk so, if we add a period at the end of out text, DataSift would see four chunks:

    The Polar Express.

Now the chunks are:

• The
• Polar
• Express
• .

DataSift it groups alphanumeric characters together but it treats punctuation characters individually, so it would divide this Tweet:

    The Polar Express...

comprises six chunks; the final three of them are simply periods:

• The
• Polar
• Express
• .
• .
• .

Punctuation is:

	Name	Symbol
	Exclamation mark	!
	Double quotes	"
	Hash	#
	Percent	%
	Ampersand	&
	Single quote	'
	Open parenthesis	(
	Close parenthesis	)
	Asterisk	*
	Plus	+
	Comma	,
	Dash	-
	Period	.
	Forward slash	/
	Colon	:
	Semicolon	;
	Less than	<
	Equals	=
	Greater than	>
	Question mark	?
	At	@
	Open square bracket	[
	Backslash	\
	Close square bracket	]
	Carat	^
	Underscore	_
	Backtick or grave accent	`
	Open curly bracket	{
	Pipe	|
	Close curly bracket	}
	Tilde	~
	Left double quote	“
	Reversed double prime quote	‵‵
	CJK reversed double prime quote	〝
	Right double quote	”
	Double prime quote	″
	CJK double prime quote	〞
	Low double prime quote	〟
	CJK ditto mark	〃
	CJK left corner bracket	「
	Left ceiling	⌈
	CJK right corner bracket	」
	Right floor	⌋
	CJK left white corner bracket	『
	CJK right white corner bracket	』

 

How does chunking affect my filters?

Suppose that you want to search for Tweets that contain "50%"; that is, the number 50 followed immediately by a percent sign. The string would be represented by two chunks:

• 50
• %

Thus, a filter such as this one:

would match "50%" but it would also match "50 %" (note the inclusion of a space) because DataSift recognizes chunks even if they have whitespace between them.

In other words, the filter would match any of these Tweets:

 

    I scored 50%

    I bought 50 %AWESOME%

    50 %25

 

So the filter would return false positives. How can we get around this problem? How can we construct a better filter? Using contains and substr together we can build a robust, reliable filter like this:

Why can't we use substr alone? Because it would return any tweet that contained 50%, 150%, 250%, and so on.