Polling in the Cloud and reliability

Posted by jack on July 11, 2015

Salesforce is a very reliable SAAS application, but that is not the case for all cloud based software – and even Salesforce is subject to occasional non-availability or time-out failures. so when designing integration allowance must be made for unexpected events and failures.

Salesforce applies API activity limits, so I want to avoid running queries too frequently, and yet still enable users to get their records updated from the data warehouse via Cast Iron in close to real time. An example I just completed involved the user inputting a reference number that creates a stub record, and the integration looking in the data warehouse for a matching record and updating the stub record with the actual values. It is possible for the user to input an invalid value (in which case a matching record is never found) or to input a reference that has not got into the date warehouse yet due to processing delays elsewhere.

The obvious approach to processing the user input is to use polling – so that a change in Salesforce is picked up at a short polling interval. This is fine provided service availability for all the components ( Salesforce, Cast Iron and the client’s data warehouse) are 100% available whenever users make the triggering change in Salesforce, and the matching record does already exist in the data warehouse  – but in practice that may not be the case, and we do have failures – one way to pick up the missed synchronisations is using a scheduled query as a backup.

So I generally couple a polling triggered orchestration with a twin “tickle” orchestration – all this does is look for records that  remain in the state that should have triggered the processing, and update them without changing any business values to cause the polling to pick them up again – that way I avoid having to replicate the business logic in two orchestrations.

Of course, to make this work  requires that the standard processing changes the processed record in a way we can detect, and also that there are some limits – generally I use 2 dateTime entries in salesforce to limit the records to reprocess 1. A “most recently last tried” limit to avoid retrying every unprocessed record  on every schedule – and 2. An “oldest record to process” limit to eliminate older records that have never processed correctly so we do eventually give up on them.





Categories: Cast Iron,Salesforce

Comments are closed.