Kafka + Redis

Overview

Modern scalable architectures decouple services and accelerate database performance using two essential distributed tools:

1. Redis: An ultra-fast, in-memory data store used for sub-millisecond caching, session state, distributed locking, and rate limiting.
2. Apache Kafka: A distributed, append-only event streaming platform used for high-throughput, fault-tolerant, asynchronous messaging across microservices.

Integrating Redis and Kafka with Spring Boot enables you to build reactive, resilient, and enterprise-grade distributed systems.

Redis

Master in-memory data structures, caching paradigms, and Spring Boot data caching integration.

Core Redis Concepts & Data Types

• What is Redis — In-memory key-value data store, single-threaded event loop architecture (I/O multiplexing), persistence mechanisms (RDB snapshots & AOF logs), and primary use cases (caching, session store, leaderboards, pub/sub).
• Redis Data Types: Strings: Basic text/binary payloads, integers, counters. Lists: Linked lists of strings. Sets: Unordered collections of unique strings. Sorted Sets (ZSet): Sets ordered by floating-point scores. Hashes: Maps of key-value pairs. Bitmaps & HyperLogLogs: Memory-efficient cardinality estimation and bit arrays.

Redis CLI Commands

• Basic Key Operations — SET key value, GET key, DEL key, EXPIRE key seconds, TTL key, EXISTS key, INCR key, DECR key.
• List Operations — LPUSH key val, RPUSH key val, LPOP key, RPOP key, LRANGE key start stop.
• Set Operations — SADD key member, SMEMBERS key, SREM key member, SISMEMBER key member, SUNION, SINTER.
• Sorted Set Operations — ZADD key score member, ZRANGE key start stop [WITHSCORES], ZREVRANGE, ZRANK, ZREM.
• Hash Operations — HSET key field value, HGET key field, HGETALL key, HDEL key field, HINCRBY key field increment.

Caching Architecture & Patterns

• Caching Patterns: Cache-Aside (Lazy Loading), Write-Through, Write-Behind (Write-Back).
• Cache Invalidation Strategies — Time-to-Live (TTL), event-driven cache eviction, cache stampede prevention (mutex locks/probabilistic early expiration), and cache penetration mitigation (Bloom filters, caching null values).
• Session Management — Centralized user session storage in Redis (spring-session-data-redis) for stateless backend horizontal scaling.
• Redis Eviction Policies — volatile-lru, allkeys-lru, volatile-lfu, allkeys-lfu, volatile-ttl, noeviction.
• Rate Limiting with Redis — Sliding window counter using Redis Sorted Sets (ZREMRANGEBYSCORE, ZADD, ZCARD) and token bucket algorithm.
• Redis Pub/Sub — Lightweight message broadcasting with PUBLISH, SUBSCRIBE, PSUBSCRIBE.

Spring Boot + Redis Integration

• Dependencies & Configuration — Adding spring-boot-starter-data-redis and configuring RedisConnectionFactory (Lettuce client).
• Spring Cache Annotations: @EnableCaching, @Cacheable, @CachePut, @CacheEvict.
• RedisTemplate Custom Operations — Configuring custom RedisTemplate<String, Object> with Jackson2JsonRedisSerializer / GenericJackson2JsonRedisSerializer for storing serialized JSON objects.

Apache Kafka

Master distributed event streaming, partitioned messaging, and enterprise Spring Kafka integration.

Core Architecture & Concepts

• Event-Driven Architecture — Decoupled asynchronous microservices, producer-consumer pattern, event sourcing, and Kafka vs traditional message queues.
• Kafka Core Components: Topic, Partition, Offset, Broker, Cluster & Controller.
• Producers — Message publishing, message keys (hashing algorithm for partition routing), acknowledgment levels (acks=0, acks=1, acks=all), and batching (linger.ms, batch.size).
• Consumers & Consumer Groups — Scaling consumption across multiple workers, consumer group rebalancing, offset commits (manual vs auto-commit), and partition assignment strategies.
• Delivery Guarantees & Exactly-Once Semantics (EOS): At-most-once, At-least-once, Exactly-once semantics (Kafka transactional API + idempotent producers).

Spring Boot + Kafka Integration

• Dependency & Configuration — spring-kafka starter, configuring bootstrap servers, producer serializer (StringSerializer, JsonSerializer), and consumer deserializer (ErrorHandlingDeserializer, JsonDeserializer).
• Sending Messages (KafkaTemplate) — Sending messages asynchronously with CompletableFuture/SendResult.
• Consuming Messages (@KafkaListener) — Listening to specific topics and groups.
• Error Handling & Dead Letter Topics (DLT) — Configuring DefaultErrorHandler, retry policies with exponential backoff, and forwarding unprocessable messages to a Dead Letter Topic (order-topic.DLT).
• Kafka Local Setup with Docker Compose — Running Kafka broker with Kraft (or Zookeeper) and Kafka UI for local inspection.

Architectural Comparison: Kafka vs Redis vs RabbitMQ