Optimized Data Synchronization with Criteria-Based Sync

You can now enable the criteria-based sync for any entity. Criteria-based sync allows the administrators to decide which records should be synced for an entity. So, there is no need to spend time syncing the customer data, which is not required. The following are the key features available with the criteria-based sync.

Ability to change the criteria at any time as the changes in filter criteria impact only the jobs that change the expiry date.
Consumer API does not need to filter the records in their search APIs (Consumers need not worry about the criteria).

Criteria-based sync uses a condition that can be applied on the turboexpirydate__c field for the given Salesforce object. The data sync does not ship the turboexpirydate__c field for the Salesforce objects out of the box, and the consumer has to create this field or populate the required objects based on their filters. The turboexpirydate__c must be created with the same name, and it is case-sensitive. The turboexpirydate__c field should be an object with 'DateTime' and 'Indexed' to pull the records from Salesforce. In addition, the turboexpirydate__c field can have three values Null, Any Future Date, and Any Past Date.

DataSync Runtime would process the pulled records (from Salesforce) in the following manner. Currently, the records have been pulled based on the 'LastModifiedDate' field.

The record is considered valid and gets synced to the consumer-endpoint if the turboexpirydate__c field value is greater than the sync-cutoff-time.
The record is considered invalid and gets deleted from the consumer-endpoint (if it exists) if the turboexpirydate__c field value is less than the sync-cutoff-time.
The record is considered ignored if the turboexpirydate__c field value is Null.

The following table provides a summary of the scenarios to understand the outcome of criteria-based sync quickly.


Scenario	Criteria-Based Sync Outcome
The value of turboexpirydate__c is null during the earlier DataSync run, but it contains a future date and time for the current run.	The record gets synced to the consumer endpoint.
The value of turboexpirydate__c was past-DateTime during the earlier DataSync-Run, but it contains a future-DateTime for the current run.	The record gets synced to the consumer-endpoint.
The value of turboexpirydate__c was future-DateTime during the earlier DataSync-Run, but it contains a past date and time for the current run.	The record gets deleted from the consumer-endpoint.
The value of turboexpirydate__c was future-DateTime during the earlier DataSync-Run, but it contains null for the current run.	The record remains unchanged at the consumer-endpoint as data sync stops tracking that record if the turboexpirydate__c has a null value. Ensure that you avoid this kind of scenario as it leads to inconsistent data in the consumer point.

Enabling criteria-based sync for an SFDC Object:

Assuming that the field turboexpirydate__c has been created and populated with the appropriate values. The following are the four possible use cases while enabling the criteria-based sync for an SFDC object.

The SFDC object is added to the customer profile for the first time.
The SFDC object is already present in the consumer profile and has gone through the Initial Sync.
The SFDC object is already present in the consumer profile and has not gone through the Initial Sync.

The SFDC object Is already present in the consumer profile and has gone through the Initial Sync using the existing filter-criteria-based sync.

S.NO.

Use Case

Recommended Actions

If the SFDC object is added to the customer profile for the first time

Invoke the following API to enable the object for criteria-based sync.


API Details	Example
POST `/ds/api/dataintegration/v1/SyncCriteria/` `{entityName}/` `enable` This API enables the criteria-based sync fr the given object across all the consumer profiles. It is not possible to enable criteria-based sync for a specific consumer profile.	POST `/ds/api/dataintegration/v1/SyncCriteria/` `{entityName}/` `enable`

If the SFDC object is already added and has gone through the Initial Sync.

You can enable this object for Criteria-based sync without cleaning the existing data (Synced data). But, both the records (previously synced and newly synced) are stored in the consumer endpoint. Therefore, it is recommended to clean up the existing data of SFDC objects from both staging-DB and consumer endpoint before enabling the SFDC object for the criteria-based sync.

To clean up the existing data from DataSync's Internal DB, invoke the following API


API Details	Example
DELETE `/ds/api/dataintegration/v1/SyncCriteria/` `{entityName}/` `CleanDataForCriteriaBasedSync`	DELETE `/ds/api/dataintegration/v1/SyncCriteria/` `{entityName}/` `CleanDataForCriteriaBasedSync`

Clean the existing data from the consumer endpoint with the help of CloudOPS.

After cleaning up the existing data from both staging DB and consumer endpoint, invoke the following API to enable criteria-based sync for the given object.


API Details	Example
POST /ds/api/dataintegration/v1/SyncCriteria/ {entityName}/enable	POST `/ds/api/dataintegration/v1/SyncCriteria/` `entityName}/` `enable`

The SFDC object is already present in the consumer profile and has not gone through the Initial-Sync

Invoke the following API to enable the object for criteria-based sync.


API Details	Example
POST `/ds/api/dataintegration/v1/SyncCriteria/` `{entityName}/` `enable`	POST`/ds/api/dataintegration/v1/SyncCriteria/` `{entityName}/` `enable`

The SFDC object Is already present in the consumer profile and has gone through the Initial-Sync using the existing filter-criteria-based sync

The filter-criteria functionality remains the same without any impact even after the upgrading to the current release builds.

Note:

This scenario is applicable only for the tenants who have enabled Filter-based sync using the earlier builds. as that works differently than the current criteria-based-sync feature.

If the existing filter-criteria is similar to the filter that is being implemented in the pre-processing job, then you can remove the filter-criteria from the object by invoking the below API.


API Details	Example
DELETE `/ds/api/dataintegration/v1/` `SyncCriteria/{consumerName}/SyncAction/{syncActionName}/FilterCriteria`	DELETE`/ds/api/dataintegration/v1/` `SyncCriteria/{consumerName}/SyncAction/{syncActionName}/FilterCriteria`

Perform all steps mentioned in the use case-2.

Disabling criteria-based sync for an SFDC Object

The following are the two possible use cases while disabling the criteria-based sync.

The object is present in the consumer profile and has not gone through the Initial Sync.

The object was earlier present in the consumer profile and has gone through the Initial Sync using Criteria Based Sync.


S.no	Use Case	Recommended Action
1	The object is present in the consumer profile and has not gone through the Initial Sync.	Invoke the following API POST`/ds/api/dataintegration/v1/SyncCriteria/{entityName}/disable`
2	The object was earlier present in the consumer profile and has gone through the Initial Sync.	Invoke the following API POST`/ds/api/dataintegration/v1/SyncCriteria/{entityName}/disable` Note: After disabling the criteria-based sync for the SFDC object, clean up the already synced data by invoking the following API. DELETE`/ds/api/dataintegration/v1/SyncCriteria/{entityName}/` `CleanDataForCriteriaBasedSync`

Implementing Criteria-Based Sync with Apex Jobs

Tenant Admin can exclude the invalid records from sync by setting the past date in the expiry date field. The expiry date is set based on the criteria condition.

Create a new turboexpirydate__c (case-sensitive) field with the appropriate date in the object on which you want to perform the Criteria-based sync.

Create a new custom metadata type.

Go to Setup > Custom Metadata Types > New Custom Metadata Type and enter the following.


Field	Input
Label	DataSyncCutoffTimeList
Visibility	Public (All Apex code and APIs can use the type, and it's visible in Setup)
Object Name	DataSyncCutoffTimeList_mdt (auto-generated)

Click Save.
A new custom metadata type field (DataSyncCutoffTimeList) page is displayed.

In the Custom fields section, click New and create the following custom fields.


Data Type	Field Lable
Text	EntityName__c
Date/Time	SyncCutoffTime__c

Create a new record as mentioned below.


Label	Product2
Name	Product2
EntityName__c	Product2
SyncCutoffTime__c	1900-01-01T12:00:00.000Z

Create a new Apex class.

Go to Setup > Apex Classes > New.

Enter the apex class code (ProductDataSyncPreProcessor2) as mentioned below.

Note:

The following Apex job code is just for reference purposes only. However, you can use this apex job code as a reference if your criteria are "Sync all products to turbo database if they have any price line item associated." For the products without Price Line Item association, the Apex job code updates the turboexpirydate__c field as the past date. Hence such products are not considered for syncing at the endpoints.

global class ProductDataSyncPreProcessor2 implements Database.Batchable<sObject> 
{
    private DateTime newCutoffDate = null;
    
    /**
     * Constructor
     */
    public ProductDataSyncPreProcessor2(DateTime newCutoff)
    {
        this.newCutoffDate = newCutoff;
    }

    /**
    * Apex batch start method
    */
    global Database.QueryLocator start(Database.BatchableContext bc) 
    {
        String strCurrentCutoffTime = newCutoffDate.formatGMT('yyyy-MM-dd\'T\'HH:mm:ss.SSS\'Z\'');

        System.debug('strCurrentCutoffTime ::: ' + strCurrentCutoffTime + '***');

        String soqlQuery =  'select Id ' +
                            'from   Product2 ' +
                            'where  turboexpirydate__c > ' + strCurrentCutoffTime + ' ' +
                            '       and Id not in ( select Apttus_Config2__ProductId__c ' +
                            '                       from    Apttus_Config2__PriceListItem__c ' +
                            '                       where   Apttus_Config2__ProductId__c != null)';

        System.debug('soqlQuery ::: ' + soqlQuery);
        return Database.getQueryLocator(soqlQuery);
    }

    /**
    * Apex batch execute method
    */
    global void execute(Database.BatchableContext BC, list<Product2> data) 
    {
        Map<Id, Product2> prodList = new Map<Id, Product2>();

        for(Product2 sObj : data) 
        {        
            Product2 prod = new Product2(Id = sObj.Id);
            prod.turboexpirydate__c = DateTime.newInstance(2000, 1, 31, 14, 0, 0);
            prodList.Put(prod.Id, prod);            
        }

        if(!prodList.IsEmpty())
            update prodList.Values();
    }
    
    global void finish(Database.BatchableContext BC) {
        setLastSyncedTime('Product2', newCutoffDate);
    }  
    
    public static void setLastSyncedTime(string objName, DateTime cutoffTime)
    {    
        List<DataSyncCutoffTimeList__mdt> historyRecs = [SELECT  EntityName__c, SyncCutoffTime__c, DeveloperName, MasterLabel
                                FROM    DataSyncCutoffTimeList__mdt
                                WHERE   EntityName__c = :objName
                                LIMIT 1]; 
                                
        if (historyRecs != null && !historyRecs.isEmpty()) {
           DataSyncCutoffTimeList__mdt dsHistory = historyRecs[0];
                     
            //create instance of Metadata.CustomMetadata
            Metadata.CustomMetadata metadataRec =  new Metadata.CustomMetadata();
            metadataRec.fullName = 'DataSyncCutoffTimeList__mdt.'+ dsHistory.DeveloperName;
            metadataRec.label = dsHistory.MasterLabel;
              
            //provide the value for the fields and add it to custom metadata instance
            Metadata.CustomMetadataValue entityNameToUpdate = new Metadata.CustomMetadataValue();
            entityNameToUpdate.field = 'EntityName__c';
            entityNameToUpdate.value = objName;
            metadataRec.values.add(entityNameToUpdate);
            
            //provide the value for the fields and add it to custom metadata instance
            Metadata.CustomMetadataValue cutoffTimetoUpdate = new Metadata.CustomMetadataValue();
            cutoffTimetoUpdate.field = 'SyncCutoffTime__c';
            cutoffTimetoUpdate.value = cutoffTime;
            metadataRec.values.add(cutoffTimetoUpdate);
              
            //Add the custom metadata instances in the container
            Metadata.DeployContainer mdContainer = new Metadata.DeployContainer();
            mdContainer.addMetadata(metadataRec);
            Id deployRequestId = Metadata.Operations.enqueueDeployment(mdContainer, null);
            System.debug('deployRequestId $$$ '+deployRequestId);
        }
    }
}

Click Save.
Go to Setup > Apex Classes > New.

Enter the apex class code (ProductDataSyncPreProcessor1) as mentioned below.

Note:

The following Apex code is just for reference purposes only.

global class ProductDataSyncPreProcessor1 implements Database.Batchable<sObject> 
{
    private DateTime newCutoffDate = System.now();
    private Boolean isInitialRun = false;
    
    /**
     * Constructor
     */
    public ProductDataSyncPreProcessor1(Boolean isFirstRun)
    {
        this.isInitialRun = isFirstRun;
    }

    /**
    * Apex batch start method
    */
    global Database.QueryLocator start(Database.BatchableContext bc) 
    {
        DateTime previousCutoffTime = getLastSyncedTime('Product2');

        String strCurrentCutoffTime = newCutoffDate.formatGMT('yyyy-MM-dd\'T\'HH:mm:ss.SSS\'Z\'');
        String strPreviousCutoffTime = previousCutoffTime.formatGMT('yyyy-MM-dd\'T\'HH:mm:ss.SSS\'Z\'');

        System.debug('strCurrentCutoffTime ::: ' + strCurrentCutoffTime + '***');
        System.debug('strPreviousCutoffTime ::: ' + strPreviousCutoffTime + '***');
        
        String soqlQuery =  'select Apttus_Config2__ProductId__c ' +
                            'from   Apttus_Config2__PriceListItem__c ' +
                            'where  (Apttus_Config2__ProductId__r.turboexpirydate__c = null or Apttus_Config2__ProductId__r.turboexpirydate__c < ' + strCurrentCutoffTime + ') ' +
                            '       and LastModifiedDate >= ' + strPreviousCutoffTime + ' ' +
                            '       and LastModifiedDate <= ' + strCurrentCutoffTime;

        System.debug('soqlQuery ::: ' + soqlQuery);
        return Database.getQueryLocator(soqlQuery);
    }

    /**
    * Apex batch execute method
    */
    global void execute(Database.BatchableContext BC, list<Apttus_Config2__PriceListItem__c> data) 
    {
        Map<Id, Product2> prodList = new Map<Id, Product2>();

        for(Apttus_Config2__PriceListItem__c sObj : data) 
        {
            if(sObj.Apttus_Config2__ProductId__c != null)
            {
                //sObj.put('Apttus_Config2__ProductId__r.turboexpirydate__c', DateTime.newInstance(2099, 1, 31, 14, 0, 0));         
                Product2 prod = new Product2(Id = sObj.Apttus_Config2__ProductId__c);
                prod.turboexpirydate__c = DateTime.newInstance(2099, 1, 31, 14, 0, 0);
                prodList.Put(prod.Id, prod);
            }
        }

        if(!prodList.IsEmpty())
            update prodList.Values();
    }
    
    global void finish(Database.BatchableContext BC) {
        System.debug('Executing ProductDataSyncPreProcessor1.Finish;'+isInitialRun+';'+newCutoffDate);
        if(isInitialRun)
        {
            setLastSyncedTime('Product2', newCutoffDate);
        }
        else
        {
            ProductDataSyncPreProcessor2 myBatchable = new ProductDataSyncPreProcessor2(newCutoffDate);    
            Database.executeBatch(myBatchable, 10000);
        }
    }  
    
    public static DateTime getLastSyncedTime(string objName){
        // Query custom metadata DataSyncCutoffTimeList__mdt as per the object name
        DateTime timeToReturn = System.now();
        DataSyncCutoffTimeList__mdt dsCutOff = null;
    
        List<DataSyncCutoffTimeList__mdt> historyRecs = [SELECT  EntityName__c, SyncCutoffTime__c
                                FROM    DataSyncCutoffTimeList__mdt
                                WHERE   EntityName__c = :objName
                                LIMIT 1]; 
                                
        if (historyRecs != null && !historyRecs.isEmpty()) {
           DataSyncCutoffTimeList__mdt dsHistory = historyRecs[0];
           if(dsHistory.SyncCutoffTime__c != null) {
              timeToReturn = dsHistory.SyncCutoffTime__c;
           }           
        }
                   
        return timeToReturn;
    }
    
    public static void setLastSyncedTime(string objName, DateTime cutoffTime)
    {    
        System.debug('Executing setLastSyncedTime;'+objName+';'+cutoffTime);
        List<DataSyncCutoffTimeList__mdt> historyRecs = [SELECT  EntityName__c, SyncCutoffTime__c, DeveloperName, MasterLabel
                                FROM    DataSyncCutoffTimeList__mdt
                                WHERE   EntityName__c = :objName
                                LIMIT 1]; 
                                
        if (historyRecs != null && !historyRecs.isEmpty()) {
           DataSyncCutoffTimeList__mdt dsHistory = historyRecs[0];
                     
            //create instance of Metadata.CustomMetadata
            Metadata.CustomMetadata metadataRec =  new Metadata.CustomMetadata();
            metadataRec.fullName = 'DataSyncCutoffTimeList__mdt.'+ dsHistory.DeveloperName;
            metadataRec.label = dsHistory.MasterLabel;
              
            //provide the value for the fields and add it to custom metadata instance
            Metadata.CustomMetadataValue entityNameToUpdate = new Metadata.CustomMetadataValue();
            entityNameToUpdate.field = 'EntityName__c';
            entityNameToUpdate.value = objName;
            metadataRec.values.add(entityNameToUpdate);
            
            //provide the value for the fields and add it to custom metadata instance
            Metadata.CustomMetadataValue cutoffTimetoUpdate = new Metadata.CustomMetadataValue();
            cutoffTimetoUpdate.field = 'SyncCutoffTime__c';
            cutoffTimetoUpdate.value = cutoffTime;
            metadataRec.values.add(cutoffTimetoUpdate);
              
            //Add the custom metadata instances in the container
            Metadata.DeployContainer mdContainer = new Metadata.DeployContainer();
            mdContainer.addMetadata(metadataRec);
            Id deployRequestId = Metadata.Operations.enqueueDeployment(mdContainer, null);
            System.debug('deployRequestId $$$ '+deployRequestId);
        }
    }
}

Note:

ProductDataSyncPreProcessor1 and ProductDataSyncPreProcessor2 are interconnected and executed one after another.
- ProductDataSyncPreProcessor1 is to update the turboexpirydate__c field of valid product records to a future date.
- ProductDataSyncPreProcessor2 is to update the turboexpirydate__c field of invalid product records to a past date.
In the first run, the ProductDataSyncPreProcessor1 is executed.
In the second run, both ProductDataSyncPreProcessor1 and ProductDataSyncPreProcessor2 are executed.

Open the developer tool and execute the following statements.

Run the following for the Initial Run:

ProductDataSyncPreProcessor1 myBatchable = new ProductDataSyncPreProcessor1(true); //Parameter 'true' is passed to indicate that this is the first run
Database.executeBatch(myBatchable, 10000);  //The second parameter can be altered based on the required batchsize.

Run the following from the second run onwards:

ProductDataSyncPreProcessor1 myBatchable = new ProductDataSyncPreProcessor1(false); //Parameter false is passed to indicate that this is not the first run
Database.executeBatch(myBatchable, 10000);  //The second parameter can be altered based on the required batchsize.

Conga Product Documentation

Welcome to the new doc site. Some of your old bookmarks will no longer work. Please use the search bar to find your desired topic.

Optimized Data Synchronization with Criteria-Based Sync

Enabling criteria-based sync for an SFDC Object:

Disabling criteria-based sync for an SFDC Object

Implementing Criteria-Based Sync with Apex Jobs

Footer Section

Explore