How to handle API Pagination in Boomi
In this Blog, we will see how to handle API Pagination in Boomi
What is API Pagination?
When we make calls to API, there will be a lot of results to return. To avoid that, we paginate the results to make sure responses are easier to handle. Pagination is when a query returns more results than the API allows to be transferred in a single response (“page”) and you must continue to make subsequent calls to request additional pages until there are no more left. This pattern is typically used with the HTTP Client or Web Services SOAP Client Connectors.
In this Use Case, we will get the API Key from NASA to configure HTTP Client Connector and get the limited set of records which also provides links to other records in response.
Let us first get the API Key from NASA.
Navigate to https://api.nasa.gov/ and sign up with your details like First Name, Last Name and Email and the API key will be generated and sent to your registered mail ID. Using this key, we will be able to access the API.
Now, we will create a process to implement the concept of API Pagination in Boomi
Step 1: Log on to the Boomi platform (https://platform.boomi.com/) with the required credentials i.e. Email Address and Password.
Step 2: Once logged into the Boomi platform, we will be able to view the Home page.
Step 3: Now, click on Services followed by Integration. We will see the Build page. Click on New.
Step 4: Once, click on New, we will be able to see three fields i.e. Type, Component Name and Folder.
- Select Type as process as we are building a process. Component Name and Folder can be given based on your choice (i.e. which name to be given and where do we want to create the process). Click on Create.
Step 5: We see that the process gets created with a start shape which is configured with AS2 Shared Server by default.
Step 6: Select the start shape and choose No Data. Click ok.
Step 7: Drag and drop set properties shape onto the process canvas to create dynamic document property and dynamic process property to hold some static values.
- Let us create by selecting the set properties shape and clicking on the + symbol as shown.
- Choose Dynamic Document Property from the drop down.
- Give the property name as “PageNumber” and click ok.
- Now, we have to set parameters for Dynamic Document Property. Select the Dynamic Document property and choose + on parameters.
- We will have to set Type and static values in parameters. Here, we are selecting Type as Static from drop down and Static Value would be 0. Click ok.
Step 8: Now we will add dynamic process property. Click on + as shown.
- Choose the Dynamic process property and add the property name as PageLimit. Click ok.
- Choose + on Parameters to set the value
- Give the static value as 9.
- Click save and close.
Step 9: Drag and drop the set properties shape and add the API key as a static value.
- Choose Dynamic Document Property from the drop-down.
- Give the property name as “Api Key” and click ok.
- Now, we have to set parameters for Dynamic Document Property. Select the Dynamic Document property and choose + on parameters.
- We will have to set Type and static values in parameters. Here, we are selecting Type as Static from the drop-down and Static Value would be the API key which we got from the NASA website. Click ok.
Step 10: Drag and drop HTTP Client onto the process canvas and configure it.
- We will have 2 actions Get and Send.
Get — If we want to get the data, we choose action as Get
Send—If we want to send the data, the action would be Send.
Step 11: Here, we choose action as Get as we are getting some data from the connector. Click + On connection and give it a name
Step 12: Add the URL of the API from which you want to get the data. Here, we are getting the data from NASA API.
Step 13: We don’t have any authentication so we will retain it to None. Click + on operation and give it a name.
Step 14: Select the response profile type as JSON as the output format for this API is in JSON Format.
Step 15: We already have the response profile which we saved in the directory. Choose the file from the saved location.
- The response profile looks like this,
Step 16: Add the rest of the URL path in the headers section i.e. click + on the resource path and add the path i.e. (neo/rest/v1/neo/browse?page=PageNumber&size=20&api_key=APIKey) as shown.
Step 17: For Page Number and API Key, put it as a replacement variable. i.e. check the box
“Is replacement variable” so that these values can be passed from the parameters section of HTTP Connector.
- Click save and close.
Step 18: Now select the connector and click on Parameters to set the values for replacement variables.
- Click on + and add the parameters
Step 19: We have two inputs which we have set as replacement variables in operation headers.
Step 20: Choose PageNumber and give Dynamic Document Property of PageNumber which we set above in set properties shape. Click ok.
Step 21: Choose APIKey and give Dynamic Document Property of API key which we have set above.
- Click ok.
Step 22: Drag and drop set properties shape to store all records in the cache.
- Choose Dynamic Document Property from the drop-down.
- Give the property name as “All” and click ok.
- We will have to set Type and static values in parameters. Here, we are selecting Type as Static from drop down and Static Value would be All. Click ok.
- Click save and close.
Step 23: We will create another Dynamic Document Property which holds the value of Total_Pages from the JSON Response profile.
- Choose Dynamic Document Property from the drop-down.
- Give the property name as Total_Pages.
- Choose the value for the property by clicking on +.
- Choose Type as Profile Element, profile Type as JSON, Profile as the JSON profile and element as Total_pages as mentioned below. Click ok.
Step 24: Drag and drop the decision shape onto the process canvas. We will check if the DDP Total pages are greater than DPP_pagelimit.
- Click on First Value choose Dynamic Document Property and give DDP Name i.e. “Total_Pages”.
- Comparison will be Greater than.
- The second value will be DPP Name i.e. “PageLimit”.
- After configuring the decision shape, it looks like
Step 25: If the condition is true, then we are setting the value of total pages to the page limit and pointing it to the branch shape. To do that, we will drag and drop set properties shape and click + on properties.
- Choose Dynamic Document Property from the drop-down.
- Choose DDP for the total pages and click ok.
- Choose a parameter by clicking on + on the parameter section and assign DPP PageLimit to it.
- Choose Dynamic process property
- Click ok.
- If the condition is false, we will point it to the branch shape.
Step 26: Select a branch shape with 2 nodes. Branch 1 will put all the records in the cache.
Step 27: Drag and drop add to cache onto process canvas.
Step 28: Select the cache and add the profile to it by clicking on + in Document Cache.
Step 29: Name the Cache add the profile type as JSON and add the profile which we have configured in the API Operation of the HTTP Client.
Step 30: Click on add index
Step 31: Add the index name as All and click ok.
Step 32: Now click on the add document property key and the Left side value will be Dynamic Document Property – All and on the right side, we will choose the DDP of All which we configured in the above steps.
- After configuring, it looks like,
- Click on save, save and close.
Step 33: Branch 2 will have the decision shape to check if the total pages are greater than the page number.
- Choose the first value as DDP of Total_Pages which is seen in the above steps.
- Comparison will be Greater than.
- The second value will be the number from the JSON response profile element.
- Once we set it, it looks like
Step 34: If the condition is false, then we will remove the records from the cache. Drag and drop “Remove From cache” and select the profile from the document cache which we have configured in add to cache.
Step 35: Choose the profile and choose the index All from the drop-down.
Step 36: Now, choose the value of Dynamic Document Property as the static value of All.
- After configuring, it looks like
Step 37: If the condition is true, we will set a DDP which will hold the current page number. Drag and drop set properties shape and click + on properties.
- Choose Dynamic Document Property from the drop down.
- Add the property name as Current_Page.
Step 38: Select + on parameters to assign the value.
Step 39: Choose Type as Profile Element, Profile Type as JSON, Profile as Json Profile which we have set in the Operation and Element as number. Click ok.
Step 40: Drag and drop data process shape to loop the records and increment the value. Select data process shape and click +.
Step 41: Choose Process Type as custom scripting from the drop down. Click on edit.
Step 42: Now, we will be able to see the command prompt as shown in the screenshot.
- Add code in the script and click ok.
Step 43: Arrange all shapes in the order and run the process by clicking on run test by configuring the atom.
Step 44: We see that the process is executed and click on data process shape and we see there are 9 records.
Step 45: If we open the first document, we see that we have the record for number 0 and has the link to next record. Next record will be number 1 (as we are incrementing the value by 1 in Data Process Shape) and so on.
Step 46: If we check the last document, it has the record of number 8 and has link to next record.
Step 47: If we click on stop shape, we can get all the records which are added to the cache once the process is executed.
