CAMEL-23239: Add camel-state-store component with pluggable key-value store

[gnodet]

JIRA: CAMEL-23239

Motivation

Camel provides dedicated components for specific caching/store technologies (Caffeine, Redis, Infinispan, etc.), each with its own API surface. This works well when users need the full feature set of a specific technology, but it creates friction when the actual need is simple: store and retrieve key-value pairs.

The camel-state-store component follows the same "choose the problem, not the technology" pattern that Camel already uses successfully in other areas:

• camel-sql / camel-jdbc — generic SQL over any JDBC database, without locking into a vendor
• camel-jms — generic messaging over any JMS provider (with camel-activemq, camel-amqp as pre-configured variants)
• camel-jcache — generic caching via JSR-107 over any compliant implementation

Similarly, camel-state-store provides a unified key-value API where:

• Users choose the capability first ("I need a key-value store") rather than a specific technology
• The backend is swappable without changing route logic — develop with in-memory, deploy with Redis or Infinispan
• The API surface is intentionally minimal (put, get, delete, contains, keys, clear) — unlike the full-featured technology-specific components

Difference from existing cache components

	camel-state-store	camel-caffeine-cache / camel-infinispan / etc.
Focus	Simple key-value store abstraction	Full feature set of a specific technology
Backend	Pluggable via StateStoreBackend interface	Fixed to one technology
API	Minimal: put, get, delete, contains, keys, clear	Rich: queries, events, statistics, pub/sub, etc.
Use case	Portability, simplicity, migration from MuleSoft Object Store	Deep integration with a specific product

Summary

• New camel-state-store component providing a simple, unified key-value store API with pluggable backends
• Supports operations: put, putIfAbsent, get, delete, contains, keys, size, clear
• Per-entry TTL with endpoint-level default and per-message override via CamelStateStoreTtl header
• Auto-discovery: if a single StateStoreBackend bean is in the registry, it is used automatically — no backend=#beanName needed on endpoints. Logs a WARN when multiple backends are found and falls back to in-memory.
• Property-based configuration: backends can be fully configured via application.properties using camel.beans.* syntax — no Java code required
• Multi-module structure with pluggable backends:
  • camel-state-store: core + in-memory backend (ConcurrentHashMap, lazy TTL)
  • camel-state-store-caffeine: Caffeine cache with per-entry variable expiry
  • camel-state-store-redis: Redisson RMapCache with native TTL
  • camel-state-store-infinispan: Hot Rod client with lifespan TTL
• Custom backends via StateStoreBackend interface and bean references
• Thread-safe backend lifecycle: start() called once per backend inside the component, idempotent guards in all backends

Route examples

Simple put/get with in-memory backend (Java DSL)

from("direct:store")
    .setHeader(StateStoreConstants.KEY, constant("user-123"))
    .to("state-store:sessions?operation=put");

from("direct:lookup")
    .setHeader(StateStoreConstants.KEY, constant("user-123"))
    .to("state-store:sessions?operation=get")
    .log("Found: ${body}");

Caching HTTP responses with TTL (Java DSL)

from("timer:poll?period=60000")
    .setHeader(StateStoreConstants.KEY, constant("weather"))
    .to("state-store:cache?operation=get")
    .choice()
        .when(body().isNull())
            .to("https://api.weather.com/current")
            .setHeader(StateStoreConstants.KEY, constant("weather"))
            .setHeader(StateStoreConstants.TTL, constant(300000L)) // 5 min TTL
            .to("state-store:cache?operation=put")
        .end()
    .to("direct:process-weather");

Idempotent deduplication pattern (Java DSL)

from("kafka:orders")
    .setHeader(StateStoreConstants.KEY, simple("${header.orderId}"))
    .to("state-store:processed?operation=putIfAbsent")
    .choice()
        .when(body().isNotNull())
            .log("Duplicate order ${header.orderId}, skipping")
            .stop()
        .end()
    .to("direct:process-order");

YAML DSL with property-configured Redis backend

# application.properties
camel.beans.redisBackend = #class:org.apache.camel.component.statestore.redis.RedisStateStoreBackend
camel.beans.redisBackend.redisUrl = redis://redis:6379
camel.beans.redisBackend.mapName = app-state

- route:
    from:
      uri: direct:save
    steps:
      - setHeader:
          name: CamelStateStoreKey
          simple: "${header.userId}"
      - to:
          uri: state-store:preferences?operation=put

- route:
    from:
      uri: direct:load
    steps:
      - setHeader:
          name: CamelStateStoreKey
          simple: "${header.userId}"
      - to:
          uri: state-store:preferences?operation=get

Dynamic operation via header

from("direct:dynamic")
    .setHeader(StateStoreConstants.KEY, simple("${header.itemKey}"))
    .setHeader(StateStoreConstants.OPERATION, simple("${header.action}"))
    .to("state-store:myStore");

Configuration examples

Java bean registration

@BindToRegistry("caffeineBackend")
public CaffeineStateStoreBackend caffeine() {
    CaffeineStateStoreBackend backend = new CaffeineStateStoreBackend();
    backend.setMaximumSize(50_000);
    return backend;
}

Property-based configuration (no Java required)

# Caffeine backend
camel.beans.caffeineBackend = #class:org.apache.camel.component.statestore.caffeine.CaffeineStateStoreBackend
camel.beans.caffeineBackend.maximumSize = 50000

# Redis backend
camel.beans.redisBackend = #class:org.apache.camel.component.statestore.redis.RedisStateStoreBackend
camel.beans.redisBackend.redisUrl = redis://myhost:6379
camel.beans.redisBackend.mapName = my-app-state

# Infinispan backend
camel.beans.infinispanBackend = #class:org.apache.camel.component.statestore.infinispan.InfinispanStateStoreBackend
camel.beans.infinispanBackend.hosts = myhost:11222
camel.beans.infinispanBackend.cacheName = my-cache

Design decision: StateStoreBackend interface location

We considered moving the StateStoreBackend interface to camel-api alongside existing SPIs (IdempotentRepository, AggregationRepository, StateRepository). We decided against it because:

• The existing camel-api SPIs are there because core EIPs consume them (idempotent consumer, aggregator). StateStoreBackend is only consumed by the component itself.
• The existing SPIs have different semantics (two-phase confirm/rollback, Exchange serialization) that don't map to a simple key-value store.
• Moving it would add API surface with stricter stability guarantees for no practical benefit today.

If a future core EIP needs a generic key-value store, the interface can be promoted then.

Test plan

• Unit tests for all operations with in-memory backend (StateStoreTest — 13 tests)
• TTL expiry tests (StateStoreTtlTest — 2 tests)
• Auto-discovery from registry (StateStoreAutoDiscoveryTest — 1 test)
• Multiple backends fallback to InMemory (StateStoreMultiBackendFallbackTest — 1 test)
• Error cases: missing key header, missing operation (StateStoreTest — 2 tests)
• Caffeine backend: all operations + TTL (CaffeineStateStoreBackendTest — 9 tests)
• Caffeine property-based config + auto-discovery (CaffeineStateStorePropertiesTest — 3 tests)
• Redis integration tests (RedisStateStoreBackendIT — 8 tests)
• Redis property-based config (RedisStateStorePropertiesIT — 2 tests)
• Infinispan integration tests (InfinispanStateStoreBackendIT — 7 tests)
• Infinispan property-based config (InfinispanStateStorePropertiesIT — 2 tests)
• Code formatted with formatter:format and impsort:sort
• Catalog metadata, Endpoint DSL, and Component DSL regenerated

[github-actions]

🌟 Thank you for your contribution to the Apache Camel project! 🌟
🤖 CI automation will test this PR automatically.

🐫 Apache Camel Committers, please review the following items:

• First-time contributors require MANUAL approval for the GitHub Actions to run
• You can use the command /component-test (camel-)component-name1 (camel-)component-name2.. to request a test from the test bot although they are normally detected and executed by CI.
• You can label PRs using build-all, build-dependents, skip-tests and test-dependents to fine-tune the checks executed by this PR.
• Build and test logs are available in the summary page. Only Apache Camel committers have access to the summary.

⚠️ Be careful when sharing logs. Review their contents before sharing them publicly.

[orpiske]

Hm, this issue doesn't seem to exist on ASF Jira.

[claudio4j]

The title refers to the wrong issue, CAMEL-23228 is "Add DataWeave to DataSonnet transpiler in camel-jbang".

[apupier]

the commit message also needs to be updated with the correct jira issue number

[apupier]

Can you elaborate on the difference between these components and the existing Camel caffeine cache component? https://camel.apache.org/components/4.18.x/caffeine-cache-component.html

[gnodet]

Thanks for the reviews! I've pushed an update:

• Fixed the JIRA reference: commit message now correctly references CAMEL-23239 (was CAMEL-23228, a copy-paste error)
• Updated the PR description with a detailed motivation section explaining the rationale and how this relates to existing components

@apupier — regarding the difference with camel-caffeine-cache: the state-store component is not a replacement for the technology-specific components. It follows the same pattern as camel-sql (generic SQL over any JDBC database) or camel-jms (generic messaging over any JMS provider). The key benefit is ease of use and backend portability — users choose the capability ("I need a key-value store") and can swap backends without changing route logic. The technology-specific components remain the right choice when users need the full feature set of a particular product (queries, pub/sub, statistics, etc.). See the updated PR description for a detailed comparison table.

Claude Code on behalf of Guillaume Nodet

[gnodet]

Design note: StateStoreBackend interface location

We considered moving the StateStoreBackend interface to camel-api so it could serve as a shared abstraction alongside the existing SPIs (IdempotentRepository, AggregationRepository, StateRepository). After analysis, we decided against it:

• The existing camel-api SPIs are there because core EIPs consume them (idempotent consumer, aggregator, etc.). StateStoreBackend is only consumed by the camel-state-store component itself — nothing in core needs it.
• The existing SPIs have different semantics that don't map cleanly to a generic key-value store: IdempotentRepository has two-phase confirm/rollback, AggregationRepository stores Exchange objects requiring serialization, and technology-specific implementations often leverage native features (JDBC transactions, Cassandra TTL, Kafka offsets).
• Moving it to camel-api would add API surface with stricter stability guarantees for no practical benefit today.

If a future core EIP needs a generic key-value store abstraction, it can be promoted then.

Claude Code on behalf of Guillaume Nodet

[Croway]

@gnodet is this kind of configuration doable only via java beans? would it be possible to provide this configuration via properties?

[gnodet]

Good point! I've pushed an update that addresses this:

Property-based configuration — backends can now be fully configured via application.properties using Camel's camel.beans.* syntax, without writing any Java code:

# Caffeine example
camel.beans.caffeineBackend = #class:org.apache.camel.component.statestore.caffeine.CaffeineStateStoreBackend
camel.beans.caffeineBackend.maximumSize = 50000

# Redis example
camel.beans.redisBackend = #class:org.apache.camel.component.statestore.redis.RedisStateStoreBackend
camel.beans.redisBackend.redisUrl = redis://myhost:6379
camel.beans.redisBackend.mapName = my-app-state

# Infinispan example
camel.beans.infinispanBackend = #class:org.apache.camel.component.statestore.infinispan.InfinispanStateStoreBackend
camel.beans.infinispanBackend.hosts = myhost:11222
camel.beans.infinispanBackend.cacheName = my-cache

Auto-discovery — if a single StateStoreBackend bean is found in the registry (whether registered via Java or via properties), it is automatically used by all state-store endpoints. No backend=#beanName reference needed:

- route:
    from:
      uri: direct:store
    steps:
      - setHeader:
          name: CamelStateStoreKey
          constant: myKey
      - to:
          uri: state-store:myStore?operation=put

Both features are covered by new tests (CaffeineStateStorePropertiesTest and StateStoreAutoDiscoveryTest).

Claude Code on behalf of Guillaume Nodet

[apupier]

maybe normal but I was surprised when reading it that when performing and operation to store a state, we change the message body with the previous value

[apupier]

I'm even wondering why modifying the body when setting something in the state store?

[gnodet]

The return-previous-value behavior follows java.util.Map.put() / Map.putIfAbsent() semantics:

• put — returns the previous value (or null if key was new), same as Map.put()
• putIfAbsent — returns the existing value if key already existed (meaning nothing was stored), or null if the value was stored successfully

This is intentional — it's the only way to communicate the result back to the route, and it enables patterns like idempotent deduplication:

from("kafka:orders")
    .setHeader(StateStoreConstants.KEY, simple("${header.orderId}"))
    .to("state-store:processed?operation=putIfAbsent")
    .choice()
        .when(body().isNotNull())  // non-null means key already existed = duplicate
            .log("Duplicate order, skipping").stop()
    .end()
    .to("direct:process-order");

This is consistent with how camel-caffeine-cache handles it (the PUT action also returns the previous value in the body).

Claude Code on behalf of Guillaume Nodet

[apupier]

Seems fine but I'm really not an expert on this piece of code and what it applies. i think it would be nice to have someone more experimented reviewing it.

[Croway]

what about using resilience4j? would it be overkill?

[Croway]

# Caffeine example
camel.beans.caffeineBackend = #class:org.apache.camel.component.statestore.caffeine.CaffeineStateStoreBackend
camel.beans.caffeineBackend.maximumSize = 50000

# Redis example
camel.beans.redisBackend = #class:org.apache.camel.component.statestore.redis.RedisStateStoreBackend
camel.beans.redisBackend.redisUrl = redis://myhost:6379
camel.beans.redisBackend.mapName = my-app-state

 # Infinispan example
camel.beans.infinispanBackend = #class:org.apache.camel.component.statestore.infinispan.InfinispanStateStoreBackend
camel.beans.infinispanBackend.hosts = myhost:11222
camel.beans.infinispanBackend.cacheName = my-cache

This sounds fine, but from a usability perspective, it would be nice to be able to reuse the existing Camel Infinispan or Redis components. This is not the first time we are "reinventing the wheel" for this kind of use case.

Given that we already have Infinispan and Redis components with their own configurations — which are supposed to be more feature-rich than the current state store implementation — I was wondering whether we could find a way to reuse those components (or at least part of their logic) for these use cases.

fyi @gnodet @orpiske @oscerd @apupier

[oscerd]

# Caffeine example
camel.beans.caffeineBackend = #class:org.apache.camel.component.statestore.caffeine.CaffeineStateStoreBackend
camel.beans.caffeineBackend.maximumSize = 50000

# Redis example
camel.beans.redisBackend = #class:org.apache.camel.component.statestore.redis.RedisStateStoreBackend
camel.beans.redisBackend.redisUrl = redis://myhost:6379
camel.beans.redisBackend.mapName = my-app-state

 # Infinispan example
camel.beans.infinispanBackend = #class:org.apache.camel.component.statestore.infinispan.InfinispanStateStoreBackend
camel.beans.infinispanBackend.hosts = myhost:11222
camel.beans.infinispanBackend.cacheName = my-cache

This sounds fine, but from a usability perspective, it would be nice to be able to reuse the existing Camel Infinispan or Redis components. This is not the first time we are "reinventing the wheel" for this kind of use case.

Given that we already have Infinispan and Redis components with their own configurations — which are supposed to be more feature-rich than the current state store implementation — I was wondering whether we could find a way to reuse those components (or at least part of their logic) for these use cases.

fyi @gnodet @orpiske @oscerd @apupier

I'm failing to see the reason why we should have another component of this kind, it will cause another dose of entropy.

[orpiske]

# Caffeine example
camel.beans.caffeineBackend = #class:org.apache.camel.component.statestore.caffeine.CaffeineStateStoreBackend
camel.beans.caffeineBackend.maximumSize = 50000

# Redis example
camel.beans.redisBackend = #class:org.apache.camel.component.statestore.redis.RedisStateStoreBackend
camel.beans.redisBackend.redisUrl = redis://myhost:6379
camel.beans.redisBackend.mapName = my-app-state

 # Infinispan example
camel.beans.infinispanBackend = #class:org.apache.camel.component.statestore.infinispan.InfinispanStateStoreBackend
camel.beans.infinispanBackend.hosts = myhost:11222
camel.beans.infinispanBackend.cacheName = my-cache

This sounds fine, but from a usability perspective, it would be nice to be able to reuse the existing Camel Infinispan or Redis components. This is not the first time we are "reinventing the wheel" for this kind of use case.

Given that we already have Infinispan and Redis components with their own configurations — which are supposed to be more feature-rich than the current state store implementation — I was wondering whether we could find a way to reuse those components (or at least part of their logic) for these use cases.

fyi @gnodet @orpiske @oscerd @apupier

Pros:

• I like the consistency
• I like that it has a swappable backend
• I like the simplicity of the API

Cons:

• We already have code that does this, even though it's inconsistent, it is well known and documented
• The existing code already fits into Camel idioms

Conclusion:

So, I am not opposed to have this one on the code base and I think there is potential for making stateful operations/caching a bit more elegant. The consistency of the operations make the behavior a bit more predictable regardless of the backend in use.

However, I believe that we should have a greater discussion about what our vision and/or medium term goals for the problem is. Even if the API is elegant, it may not be enough if we don't collectively share the same vision about how we would like this to be ...

As such, I'm currently leaning towards 0 on this one and would like to hear more feedback from the community before having a decision on -1 or +1 from my side.