Prerequisites:

Anypoint Studio:

• Anypoint Studio should be set up.

Azure Portal:

• Azure Account:

Step1: Go to https://portal.azure.com , click Create a free account, and sign in with your Microsoft credentials (or create one if you don’t have it).Follow the on-screen instructions to verify your identity, provide subscription details (like payment method for verification), and complete the account setup.

Step2: In the Azure Portal, go to Create a resource → search for Storage account → click Create, then fill in the basics (subscription, resource group, storage account name, region, performance, and redundancy).

Step3: In the advance tab enable Hierarchical Namespace to use it as Azure Data Lake Storage Gen2.

Step4: Go to Azure Active Directory → App registrations → click new registration, enter a name for your app, choose who can use it (single tenant or multi-tenant), then click Register.

Step5: From the app’s Overview page, copy the Application (client) ID and Directory (tenant) ID — these are needed for authentication. Go to Certificates & secrets → New client secret, enter a description and expiry period → click Add, then copy the generated Value (this is the client secret).

Step6: Navigate to your Storage Account → Access Control (IAM) → click Add → Add role assignment. In the Role tab, search for “Storage Blob Data Contributor”, select it (this role allows read, write, and delete access to blob containers and data) → click Next. In the Members tab → click Select members, pick your registered app (service principal) → click Select → Next.

Azure Data Lake Storage Connector in MuleSoft

Overview:

• In today’s data-driven world, organizations rely on cloud-based solutions to store, manage, and analyse vast amounts of data. Azure Data Lake Storage (ADLS) is Microsoft’s scalable, secure, and cost-effective data lake service designed to handle massive volumes of structured and unstructured data. MuleSoft, on the other hand, is a leading integration and API management platform that enables enterprises to connect systems, applications, and data sources seamlessly.
• Bringing these two powerful technologies together allows enterprises to create automated, real-time, and reusable data pipelines. With MuleSoft’s API-led approach and Azure’s cloud-based storage capabilities, businesses can centralize their data, power analytics, and enhance decision-making while maintaining flexibility and security across the integration layer.

What is Azure Data Lake Storage?

• Azure Data Lake Storage (ADLS) is a cloud-based data lake solution built to store, manage, and analyze large datasets of any type or size. It is optimized for high-throughput and parallel processing, making it ideal for big data analytics, artificial intelligence, and machine learning workloads. ADLS supports hierarchical namespace, which allows organizations to manage data with file and folder semantics for better organization and performance.
• Some of its core capabilities include:
  • Massive scalability for handling petabytes of data.
  • Native integration with Azure analytics tools such as Synapse, Databricks, HDInsight, and Power BI.
  • Fine-grained access control and security features integrated with Azure Active Directory.
  • Cost-effectiveness with pay-as-you-go pricing and tiered storage options.

Why Use MuleSoft with ADLS?

MuleSoft excels at connecting different systems, data sources, and applications across hybrid environments. When integrated with ADLS, MuleSoft allows organizations to:

• Automate data ingestion and transformation from various enterprise systems to Azure Data Lake.
• Expose ADLS data as APIs for consumption by internal teams, analytics platforms, or external partners.
• Enable real-time or batch-based processing by orchestrating workflows across cloud and on-premises environments.
• Maintain governance, security, and compliance through MuleSoft’s centralized integration architecture.

This integration is especially valuable in enterprise ecosystems where multiple data sources (e.g., CRM, ERP, marketing platforms) need to feed into a central data lake for unified analytics and reporting.

Understanding Key Components of the Azure Data Lake Storage Connector in MuleSoft

• MuleSoft provides the following essential components within its Azure Data Lake Storage Connector to simplify and streamline integration with ADLS Gen2:
• 
• Create File System: Creates a new file System (container) within the Azure Data Lake Storage account.
• List File Systems: Retrieves a list of all file systems in the storage account.
• Create/Rename Path: Creates a new file or directory, or renames an existing one within a file system
• Delete File System: Deletes an entire file system from the storage account.
• Delete Path: Deletes a specified file or directory inside a file system
• Get Path Properties: Retrieves metadata, properties, or access control (ACL) for a given file or directory.
• Create File System In Azure:
• The Create File System operation in MuleSoft’s Azure Data Lake Storage connector allows you to create a new file system (container) in your Azure Data Lake Storage account. A file system acts as a logical container to store and organize your data (files and directories) and is the first step before uploading or processing any data in ADLS.
• Steps:

1. To begin, a new Mule project needs to be created. Then, a HTTP listener component dragged and dropped into the project from the HTTP module. Configure the listener and set the host to All interface (0.0.0.0), port number as 8081. Set the path as /fileSystem.

1. Following the addition of the listener component, include two loggers to mark the beginning and the end of the flow. By logging the message “Start of flow” ++ (flow.name as String), in expression mode.

1. Within the Mule Palette, use the search function in the exchange to look for the Azure Data Lake Storage connector and add it.

1. After adding the Connector, configure it by choosing OAuth 2.0 Connection Provider to enable secure authentication with Azure Active Directory.

• Base URI → https://<your-storage-account-name>.dfs.core.windows.net
• Client ID → Use the Application (client) ID from your Azure App Registration.
• Client Secret → Use the Secret Value generated under Certificates & Secrets.
• Token URL → https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token (replace <tenant-id> with your Directory (tenant) ID).
• Scopes → https://storage.azure.com/.default

1. Following the addition of the initial logger, use a Set Variable component to extract the fileSystem value from the HTTP request headers by setting its value to #[attributes. headers.’fileSystem’] and storing it in a flow variable named filesystem.

• Drag and drop the Azure Create File System component into the flow, pass the previously set filesystem variable as the value for the filesystem field, and configure the timeout property (e.g., 60 seconds) to create the specified file system in Azure Data Lake Storage.
• 
• Provide a request to Api and response will be provide mentioned below.
• 
• Go to your Azure Storage Account in the portal, navigate to Containers, and verify that the newly created container (e.g., demo-container-poc) is listed with the expected access level and lease state.
• 
• Create/rename path:
• The Create/Rename Path operation in MuleSoft’s Azure Data Lake Storage connector enables you to create new directories or rename existing paths within your Azure Data Lake Storage account. This operation helps you structure and manage your data hierarchy efficiently, ensuring that files and folders are logically organized and easily accessible for downstream processing or analytics.
• Steps:
• Configure the listener and set the host to All interface (0.0.0.0), port number as 8081. Set the path as / create Path.
• 

1. Use another Set Variable component to extract the path value from the HTTP request headers by setting its value to # [attributes. headers. ’path’] and storing it in a flow variable named path along with filesystem.

1. Drag and drop the Azure Create/Rename Path component into the flow, pass the previously set filesystem and path variables as the values for the File System and Path fields, select the Resource type as file (because we are inserting a file into that directory.       
2. Hit the request on the postman to the Api and response will be provided as below.

• Go to your Azure Storage Account in the portal, navigate to Containers, and verify that the newly created file (e.g., sample. Json) in the container (demo-container-poc) is listed with the expected access level and lease state.

• List File Systems:
• The List File Systems operation in MuleSoft’s Azure Data Lake Storage connector allows you to retrieve all existing file systems (containers) within your Azure Data Lake Storage account. This operation helps you view and manage the logical containers that store and organize your data (files and directories). It is often used as a verification step to confirm the availability of file systems before performing further data operations, such as creating paths, uploading files, or managing directories.
• Steps:

1. Configure the listener and set the host to All interface (0.0.0.0), port number as 8081. Set the path as /list.

• 2. Drag and drop the Azure Create File System component into the flow, pass the previously set filesystem variable as the value for the filesystem field, and configure the timeout property (e.g., 60 seconds) to create the specified file system in Azure Data Lake Storage.
• 
• 3. Provide a request to Api and response will be provide mentioned below.
• 

Modernize your data management and unlock seamless cloud integrations. Connect with TGH’s MuleSoft experts to implement secure, scalable, and automated Azure Data Lake solutions that deliver real business insights!