Identifying Intent and Desire

One of the quickest ways to add value to data in PYLON is classification using tags.

In this simple example we'll take a look at building a classifier which identifies desire expressed in content. By identifying desire with a tag you can then use this as a filtering option when submitting analysis queries to your index.

tip icon

This simple example uses keywords to identify intent. Another approach you could take would be to apply machine-learning results.

Our developer guide gives you an introduction to the concepts of classification.

Identifying Keywords and Phrases

When you create a classifier that identifies intent or emotion it is imporant to focus on a specific domain. It's extremely difficult to create an accurate classifier that will work across all domains. For this example we focused on people discussing movies, essentially we were looking to identify people expressing an desire to see a movie.

As you'll be aware by now, the privacy model of PYLON prevents you from accessing raw data from conversations. This gives you a challenge when trying to identify keywords for your tags.

To generate keywords for this classiser we took a range of sources - including social networks and review sites. We then examined the words and phrases used by people expressing a desire to see a movie. This can be a time consuming manual process, but it is worth the investment to produce a high quality classifier.

Building the Classifier

For this classifier we want to classify desire, so we need to build a tag for this class. The classifier needs to tag both stories and engagements on those stories.

The result of our earlier analysis of various sources produced two keyword lists:

  • Include - keywords we wanted to include - "i want this,just wanna go,need someone to, ..."
  • Exclude - keywords we wanted to exlude - "don't need,didn't want a,doesn't need to, ..."

The tag can be built with the following syntax:

tag "Desire" { 
     ( fb.content any "[include keywords]" 
    AND NOT fb.content any "[exclude keywords]" )
    OR 
    ( fb.parent.content any "[include keywords]" 
    AND NOT fb.parent.content any "[exclude keywords]" )
}

Looking at the syntax in detail:

  • tag - declares this this as a tag rule
  • "Desire" - is the label for the class
  • fb.content any - looks for keywords in Facebook stories (any is shorthand for the contains_any operator)
  • fb.parent.content any - looks for keywords in stories that are engaged with

Adding our keywords gives the final tag:

tag "Desire" { 
     ( 
         fb.content any "i want this,just wanna go,need someone to,wanna be a,want to buy,want to let,we need more,need to start,wants a new,need to make,i can be,i wanna do,we need a,i want them,just need to,i need more,i want the,they want to,want to talk,i really wanna,need this in,need someone who,wants to go,i want is,want to know,want me to,need to see,just wanna be,need a new,makes me want,need to do,need to stop,i either want,or i want,i need someone,i just need,in need of,all i need,i wish my,what i want,i really need,i want my,really need to,me want to,emotion i need,i need some,i wanted to,want to do,i wanna be,really want to,wants to be,i really want,want to see,want to get,wanna go to,wish i was,all i want,wish i had,want to spend,i want some,need to be,we need to,need to get,i wanna go,wish i could,just want to,want to be,i just wanna,i want a,i need this,want to go,i wish i,i just want,i need a,i need to,i want to" 
         AND NOT fb.content any "don't need,didn't want a,doesn't need to,don't wanna stay,don't want anyone,don't want nobody,i didn't wanna,never wanna leave,never wanted to,that don't want,wouldn't want to,didn't need to,doesn't wanna be,doesn't want me,do not wanna,don't even need,don't need no,don't wanna wake,don't want him,#scorpio don't need,who doesn't wanna,who don't wanna,but not needy,don't need any,don't need anyone,don't need nobody,don't wanna have,don't wanna leave,don't wanna talk,don't want my,i never want,not needy by,who don't need,don't want them,don't wanna do,don't want it,not wanting to,we don't want,didn't want me,don't need someone,don't want no,they don't want,just don't want,don't want a,just don't wanna,wish i didn't,don't even wanna,never want to,we don't need,doesn't want to,nigga don't wanna,don't even want,don't wanna get,don't wanna give,really don't wanna,do not want,not want to,really don't want,i didn't want,brother don't want,don't need to,don't wanna go,didn't want to,don't need a,don't wanna be,i don't need,i don't wanna,i don't want,don't want to" 
     ) 
     OR 
     ( 
         fb.parent.content any "i want this,just wanna go,need someone to,wanna be a,want to buy,want to let,we need more,need to start,wants a new,need to make,i can be,i wanna do,we need a,i want them,just need to,i need more,i want the,they want to,want to talk,i really wanna,need this in,need someone who,wants to go,i want is,want to know,want me to,need to see,just wanna be,need a new,makes me want,need to do,need to stop,i either want,or i want,i need someone,i just need,in need of,all i need,i wish my,what i want,i really need,i want my,really need to,me want to,emotion i need,i need some,i wanted to,want to do,i wanna be,really want to,wants to be,i really want,want to see,want to get,wanna go to,wish i was,all i want,wish i had,want to spend,i want some,need to be,we need to,need to get,i wanna go,wish i could,just want to,want to be,i just wanna,i want a,i need this,want to go,i wish i,i just want,i need a,i need to,i want to" 
         AND NOT fb.parent.content any "don't need,didn't want a,doesn't need to,don't wanna stay,don't want anyone,don't want nobody,i didn't wanna,never wanna leave,never wanted to,that don't want,wouldn't want to,didn't need to,doesn't wanna be,doesn't want me,do not wanna,don't even need,don't need no,don't wanna wake,don't want him,#scorpio don't need,who doesn't wanna,who don't wanna,but not needy,don't need any,don't need anyone,don't need nobody,don't wanna have,don't wanna leave,don't wanna talk,don't want my,i never want,not needy by,who don't need,don't want them,don't wanna do,don't want it,not wanting to,we don't want,didn't want me,don't need someone,don't want no,they don't want,just don't want,don't want a,just don't wanna,wish i didn't,don't even wanna,never want to,we don't need,doesn't want to,nigga don't wanna,don't even want,don't wanna get,don't wanna give,really don't wanna,do not want,not want to,really don't want,i didn't want,brother don't want,don't need to,don't wanna go,didn't want to,don't need a,don't wanna be,i don't need,i don't wanna,i don't want,don't want to" 
     ) 
}

You can see the final classifier in the library.

Applying the Classifier to a Recording

It's easy to apply a set of tags to an interaction filter and create a recording.

For this example let's say we have an interaction filter already defined with the following CSDL:

return { 
    fb.topics.category in "Movie,Film,TV Network,TV Programme,TV Channel,TV Show,TV/Movie Award" 
    OR fb.parent.topics.category in "Movie,Film,TV Network,TV Programme,TV Channel,TV Show,TV/Movie Award" 
}

Firstly you need to ensure your filter conditions are encapsulated in a return statement. A return statement is mandatory when using tags in a filter.

Then you can include the tag, either by adding them before the return statement:

tag "Desire" { 
     ( 
         fb.content any "i want this,just wanna go,need someone to,wanna be a,want to buy,want to let,we need more,need to start,wants a new,need to make,i can be,i wanna do,we need a,i want them,just need to,i need more,i want the,they want to,want to talk,i really wanna,need this in,need someone who,wants to go,i want is,want to know,want me to,need to see,just wanna be,need a new,makes me want,need to do,need to stop,i either want,or i want,i need someone,i just need,in need of,all i need,i wish my,what i want,i really need,i want my,really need to,me want to,emotion i need,i need some,i wanted to,want to do,i wanna be,really want to,wants to be,i really want,want to see,want to get,wanna go to,wish i was,all i want,wish i had,want to spend,i want some,need to be,we need to,need to get,i wanna go,wish i could,just want to,want to be,i just wanna,i want a,i need this,want to go,i wish i,i just want,i need a,i need to,i want to" 
         AND NOT fb.content any "don't need,didn't want a,doesn't need to,don't wanna stay,don't want anyone,don't want nobody,i didn't wanna,never wanna leave,never wanted to,that don't want,wouldn't want to,didn't need to,doesn't wanna be,doesn't want me,do not wanna,don't even need,don't need no,don't wanna wake,don't want him,#scorpio don't need,who doesn't wanna,who don't wanna,but not needy,don't need any,don't need anyone,don't need nobody,don't wanna have,don't wanna leave,don't wanna talk,don't want my,i never want,not needy by,who don't need,don't want them,don't wanna do,don't want it,not wanting to,we don't want,didn't want me,don't need someone,don't want no,they don't want,just don't want,don't want a,just don't wanna,wish i didn't,don't even wanna,never want to,we don't need,doesn't want to,nigga don't wanna,don't even want,don't wanna get,don't wanna give,really don't wanna,do not want,not want to,really don't want,i didn't want,brother don't want,don't need to,don't wanna go,didn't want to,don't need a,don't wanna be,i don't need,i don't wanna,i don't want,don't want to" 
     ) 
     OR 
     ( 
         fb.parent.content any "i want this,just wanna go,need someone to,wanna be a,want to buy,want to let,we need more,need to start,wants a new,need to make,i can be,i wanna do,we need a,i want them,just need to,i need more,i want the,they want to,want to talk,i really wanna,need this in,need someone who,wants to go,i want is,want to know,want me to,need to see,just wanna be,need a new,makes me want,need to do,need to stop,i either want,or i want,i need someone,i just need,in need of,all i need,i wish my,what i want,i really need,i want my,really need to,me want to,emotion i need,i need some,i wanted to,want to do,i wanna be,really want to,wants to be,i really want,want to see,want to get,wanna go to,wish i was,all i want,wish i had,want to spend,i want some,need to be,we need to,need to get,i wanna go,wish i could,just want to,want to be,i just wanna,i want a,i need this,want to go,i wish i,i just want,i need a,i need to,i want to" 
         AND NOT fb.parent.content any "don't need,didn't want a,doesn't need to,don't wanna stay,don't want anyone,don't want nobody,i didn't wanna,never wanna leave,never wanted to,that don't want,wouldn't want to,didn't need to,doesn't wanna be,doesn't want me,do not wanna,don't even need,don't need no,don't wanna wake,don't want him,#scorpio don't need,who doesn't wanna,who don't wanna,but not needy,don't need any,don't need anyone,don't need nobody,don't wanna have,don't wanna leave,don't wanna talk,don't want my,i never want,not needy by,who don't need,don't want them,don't wanna do,don't want it,not wanting to,we don't want,didn't want me,don't need someone,don't want no,they don't want,just don't want,don't want a,just don't wanna,wish i didn't,don't even wanna,never want to,we don't need,doesn't want to,nigga don't wanna,don't even want,don't wanna get,don't wanna give,really don't wanna,do not want,not want to,really don't want,i didn't want,brother don't want,don't need to,don't wanna go,didn't want to,don't need a,don't wanna be,i don't need,i don't wanna,i don't want,don't want to" 
     ) 
}

return { 
    fb.topics.category in "Movie,Film,TV Network,TV Programme,TV Channel,TV Show,TV/Movie Award" 
    OR fb.parent.topics.category in "Movie,Film,TV Network,TV Programme,TV Channel,TV Show,TV/Movie Award" 
}

Or to make your code more maintainable, we recommend saving your tags definition as a filter, then including these in your filters using the tags keyword.

Analyzing Classified Data

Once you've recorded data based upon a classifier you can make use of the tags in your analysis queries.

In this case you could filter to just conversations showing 'desire' by specifying the following as your filter parameter:

interaction.tags == "Desire"

Of course you can then add further filter conditions and dig down by demographic or other tags you choose to add to the data.