Blog Post

Introduction

What is an Object Store?

An object store is a mechanism for storing objects within or across Mule applications. The Mule runtime engine (Mule) leverages object stores to persist data for future retrieval. Internally, Mule uses object stores in various filters, routers, and message processors that require state persistence between messages.

Where are object stores used?

• In the Idempotent Message Validator, unique IDs get stored in the object store.
• In the Redelivery Policy, the redelivery count gets stored in the object store.
• In the Cache Scope with a custom caching strategy, the cached data gets stored in the object store.
• In a custom component, data gets stored in the object store for persistence.
• In the Anypoint Object Store Connector, information gets stored or retrieved from the object store within a Mule flow example OAuth tokens.

What are the different types of object stores?

The three types of Object Store are:

• Anypoint Object Store V2
• CloudHub Object Store V1
• Mule Object Store

What are the different instances of object store?

There are three instances of Object Store

• Default Object store
• Global Object Store
• Private Object Store

 Object Store Configuration Options

• Entry TTL (Time to Live): This is the lifespan of an entry in the object store. If not specified, there is no time limit for how long the entry will remain in the store.
• Entry TTL Unit: Defines the unit of time (seconds, minutes, hours, etc.) for the TTL value.
• Expiration Interval: This specifies how frequently the object store checks for expired entries. Elements that exceed their TTL or the max entries will be discarded. If neither TTL nor max entries are defined, this has no effect.
• Expiration Interval Unit: Defines the unit of time (seconds, minutes, hours, etc.) for the expiration interval.
• Persistent: Determines whether the data should be stored on disk (true) or kept in memory (false). Persistent storage ensures data is saved across application restarts.
• Max Entries: This sets the maximum number of entries that can be stored. If the number exceeds the limit, older entries are removed when the expiration thread runs. If not set, there is no limit on the number of entries.
• Persistence Location: If persistence is enabled, the object store data is saved in the file system at <MULE_HOME>/.mule/<<app>>/objectstore/, ensuring it survives restarts and crashes.

Anypoint Object Store V2 (OSv2)

Object Store V2 is the latest cloud-based storage solution for CloudHub applications, offering scalable key-value storage. It’s designed to replace the older Object Store V1, providing better performance and expanded functionality.

Key Characteristics:

• Time to Live (TTL): Default TTL of 30 days. If accessed in the last seven days of TTL, the data extends by another 30 days. Some Mule 4 runtimes allow configurable TTL.
• Unlimited Storage Capacity: While there’s no overall limit on the number of entries, each individual value can be up to 10 MB in size.
• Key Length: Maximum of 256 characters for each key.
• Security: TLS encryption with FIPS 140-2 compliance for persistent storage.
• Connector Integration: The Object Store Connector supports OSv2, but certain attributes (e.g., Expiration Interval, Max Entries) are not configurable.
• API Rate Limits: Standard service allows 10 TPS; the premium version offers up to 100 TPS.

Common Applications:

• Data Persistence: Suitable for CloudHub applications that need simple, scalable key-value storage.
• Data Sharing: Used for sharing data between multiple workers in a multi-worker CloudHub deployment, though concurrency limitations may apply.

Important Considerations:

• Region-Specific Storage: OSv2 ensures data is stored in the same region as the CloudHub application, minimizing latency.
• API Transaction Limits: Consider using the premium version if your application demands more than the 10 TPS offered by the standard plan.
• Mule Runtime Compatibility: OSv2 is mandatory for Mule Runtime versions 4.x and above.

CloudHub Object Store V1 (OSv1)

OSv1 is the original Object Store for CloudHub but has been deprecated in favor of OSv2. It offers basic key-value storage for applications deployed on CloudHub but will eventually reach its End-of-Life.

Key Characteristics:

• Key and Value Limits: Up to 100,000 key-value pairs per application. Keys can be up to 768 bytes, and individual values are capped at 1 MB.
• Total Data Capacity: Limited to 1 GB of data per application.
• Connector Integration: Supports the Object Store Connector, but options like TTL, Expiration Interval, and Max Entries are not configurable.

Common Applications:

• Basic Data Storage: Ideal for simple key-value storage in CloudHub applications that do not require high storage capacity.
• Multi-Worker Data Sharing: Can be used to share data between multiple workers in a multi-worker (fabric) CloudHub deployment.

Important Considerations:

• Deprecation: OSv1 is deprecated, and users should migrate to OSv2 for long-term support.
• Runtime Compatibility: OSv1 is not supported by Mule Runtime 4.x. Transition to OSv2 is required for compatibility with newer Mule runtimes.
• Data Limitations: OSv1’s storage capacity is significantly smaller compared to OSv2, making it less suitable for larger-scale applications.

Mule Object Store (On-Premises)

The Mule Object Store is a flexible, customizable solution available as part of the Mule Runtime. It is used in on-premises or hybrid environments and provides configurable options for both in-memory and persistent storage.

Key Characteristics:

• Customizable Storage: No fixed limits on key size, value size, or the total size of the object store. Attributes like Max Entries, Entry TTL, and Expiration Interval are fully configurable.
• In-Memory vs. Persistent: Can store data either in-memory for fast access or persist it to disk for long-term retention.
• Storage Location: For persistent storage, the default path is <MULE_HOME>/.mule/<<app>>/objectstore/.

Common Applications:

• On-Premises and Hybrid Deployments: Suitable for standalone Mule Runtime deployments or those in a high-availability cluster. For HA Clusters, the object store should be configured in-memory to allow data replication across nodes.
• Cross-Flow Data Persistence: Allows Mule applications to store and reference data across multiple flows and events.

Important Considerations:

• Resource Management: If not configured correctly, in-memory stores may exhaust JVM heap space, and disk-based stores could use up disk space, potentially leading to crashes. Proper limits on Max Entries and TTL should be set.
• Data Loss on Redeployment: In cases where persistent data is lost after redeployment, a patch (SE-12341) for Mule Runtime 4.x may be required.
• Runtime Fabric Limitations: Persistent Object Store is not supported on Runtime Fabric without a configured “Persistence Gateway” due to Kubernetes’ behavior of destroying and recreating Mule nodes (pods), which can result in data loss. However, data can be retained in-memory by deploying the application in “clustered” mode with multiple replicas and employing a rolling update strategy to ensure the transfer of Object Store key-values during redeployment.

Default Object store:

The Default Object Store in MuleSoft is automatically created by Mule Runtime for storing temporary data. We can not reuse the default object store because we explicitly don’t create a configuration.

Drawbacks

• We cannot reuse the object store partition.
• We cannot set custom TTL

As you can see from the above picture, no configuration has been added. If we deploy this application, it will automatically create a default partition and store the data in it, as this is the default behavior provided by MuleSoft.

Global Object Store:

The Global Object Store in MuleSoft is a custom-defined, reusable object store configuration that can be shared across different flows and components within a Mule application. It is configurable for persistence, entry time-to-live (TTL), and other attributes. The global object store allows centralized data storage, making it accessible from anywhere in the application.

Advantages:

• The global object store can be shared across different flows and components within a Mule application.
• We can create different partitions of object store.

Disadvantage:

• Any other components can make changes to the particular configuration/partition data.

If we create multiple configurations and store different data under each one, a separate partition will be created for each configuration, and the data will be stored accordingly.

Private Object Store:

The Private Object Store in MuleSoft is an object store that is automatically created for specific components and is accessible only within the scope of that component. It cannot be shared across different flows or components.

Advantages:

• Remains private to a particular component only.
• Have all the customization features as a Global object store(TTL, Max Entries, persistent).

Disadvantages:

• Can’t be accessed outside of a component.
• It is only available in some components ex Cache.

From the above images, you can see that by adding a caching strategy and selecting ‘Edit Inline’ in the Object Store reference, you can create a configuration for a private object store.