Scatter-Gather

MuleSoft

_ August 31, 2024_ Aniket Pal

Scatter-Gather

Pre-requisites

Any point Studio:

Any point Studio should be set up.

Salesforce:

We need to have a Salesforce developer account.
Create a Salesforce custom object with some fields, as per the requirement.
Click on Create in the home section, which is on the right-hand side. Then click on a custom object, then fill in the name of the custom object. Then click on fields and relationship, then select the fields with the respective type of information.

In this scenario, we’ll be having 4 fields i.e. Id__c, Name__c, Designation__c, Contact, Age__c and all the fields have the text as the type of information stored in it.

We need to keep the security token handy for that, click on profile and then click on setting which is on the right-hand top corner of the screen type security token in the quick find section, and click on reset security token.
- Note: Once you have a security Token you can reuse the same as if the token is again reset the integrations that rely up on the old token don’t work if a new token is created unless it is changed.
The security token will be sent to the registered mail ID.

Database:

We need to have a database.
For this instance, we’ll be using the MySQL database.
Create a database with some name and inside the database create a table with some fields as per the requirement.
For this instance, we’ll be having the table name as customer_details and the fields are EmpId, EmpName, Contact, Designation and Age, where EmpId is the primary key.

Scatter-Gather Router

Scatter-gather router is part of the core module, which falls under the category of flow control.
It is the component that is used for achieving parallel processing in MuleSoft.
Scatter-gather executes each route in a different thread and returns the consolidated mule event from all routes.
It has 3 properties which can be configured if needed:
- Timeout: Timeout means how much time to wait before the route throws a timeout error.
- Target: Target means the name of the target variable, where the result of scatter-gather will get stored and can be referenced later in the flow.
- Max Concurrency: Max Concurrency defines the maximum number of concurrent threads that can process the message in the flow.
This component returns the HashMap as a result.
The Scatter-gather component will throw a composite routing error if one of the route errors out.
It should have a minimum of two routes, we’ll get an m-unit error and the application will not get deployed.

In this blog, we’ll see how to achieve parallel processing in MuleSoft. The requirement is to read the data from the xlsx file store it to the Salesforce custom object and DB.
For this requirement, we’ll be using MySQL db.

Steps:

To begin, a new Mule project needs to be created. Then, a HTTP listener component dragged and dropped into the project from the HTTP module. Configure the listener, and set the host to All interface (0.0.0.0), port number as 8081. Set the path as /r.

2. Following the addition of the listener component, include two loggers to mark the beginning and the end of the flow. By logging the message “Start of flow” ++ (flow.name as String), in expression mode.

3. Now we’ll add read component by configure the connector configuration with parent directory & file path as the name of the file and MIME type as application/xlsx .

4. Next, we’ll add the transform the message to transform the data into JSON format, for that we have to setup the metadata of the incoming payload with Sheet number as well as the column name same as in xlsx file.

5. Then we’ll add Scatter-Gather and configure the first route by adding a bulk insert. This is an upsert query for MySQL db. We’ll be placing a transform message to map the records to the database.