Dynamic Push Directories for File-Based Connectors

For the FTP, SFTP and Amazon S3 file-based connectors you can specify all or part of the path dynamically for output_params.directory for these endpoints:

For example, if {d} represents the day of the month, your path could be:

/tests/test01/{d}

For Thursday August 19, 2015, your data would be written to a directory called:

/tests/test01/19

Here are the parameters you can use:

UTC time
{U} UTC timestamp
Day
{d} Day of the month, 2 digits with leading zeros 01 - 31
{j} Day of the month without leading zeros 1 - 31
{D} A textual representation of a day, three letters Mon - Sun
{l} A full textual representation of the day of the week Monday - Sunday
Month
{F} A full textual representation of a month, such as January or March January - December
{M} A short textual representation of a month, three letters Jan - Dec
{m} Numeric representation of a month, with leading zeros 01 - 12
{n} Numeric representation of a month, without leading zeros 1 - 12
Year
{Y} A full numeric representation of a year, 4 digits For example: 1999 or 2003
{y} A two digit representation of a year For example: 99 or 03
Hour
{g} 12-hour format of an hour without leading zeros 1 - 12
{G} 24-hour format of an hour without leading zeros 0 - 23
{h} 12-hour format of an hour with leading zeros 01 - 12
{H} 24-hour format of an hour with leading zeros 00 - 23
Minute
{i} Minutes with leading zeros 00 - 59
Subscription
{id} Subscription ID
{type} Type stream or historic
{stream_hash} The stream hash
{historics_id} The playback ID
{hash} The stream hash or playback id, depending on which is present

Here's an example using FTP:

curl -X POST 'https://api.datasift.com/v1/push/create' \
-d 'name=connectorftp' \
-d 'hash=42d388f8b1db997faaf7dab487f11290' \
-d 'output_type=ftp' \
-d 'output_params.host=ftp.example.com' \
-d 'output_params.port=21' \
-d 'output_params.auth.username=dsreceiver' \
-d 'output_params.auth.password=dspassword' \
-d 'output_params.directory=%2Fhome%2Fdsreceiver%2F%7Bm%7D%2F%7Bd%7D' \
-d 'output_params.delivery_frequency=60' \
-d 'output_params.max_size=10485760' \
-d 'output_params.mark_in_progress=true' \
-H 'Authorization: datasift-user:your-datasift-api-key'

Note that:

  1. All timestamps are specified in UTC.
  2. DataSift can create directories for you as long as the username you specify has those privileges.
  3. We recommend that you use at least one static parent directory in your path.