You use the output_params.delivery_frequency parameter to specify how often DataSift checks its internal queue to see if there is any data waiting for delivery. If you set this value to zero, we will recheck the queue continuously.
Each connector has a throughput limit. Let's go through an example using the HTTP connector, which has a data delivery limit of 20MB:
- DataSift checks the internal queue to see if your stream has produced any data and it delivers all the data it finds, up to the 20MB limit.
- Once the data has been delivered, we check the queue again immediately. If there is at least 20MB of data still in the queue, we deliver another 20MB immediately. If there is less than 20MB, we leave it in the queue.
- We continue this process, for up to five deliveries of 20MB, making 100MB in total.
- Then, as quickly as possible, we go back to step 1 and start the process all over again.
Remember, some connectors have a smaller limit. For instance, our limit for CouchDB is 5MB.
Please accept our apologies for hiding a GOTO statement in the above list.