Design Distributed Cache | System Design

Designing a Distributed Cache system requires careful consideration of scalability, fault tolerance, and performance. This article explores key architectural decisions and implementation strategies to create an efficient, high-performance caching solution.

What is caching?

In computing, a cache is a high-speed data storage layer that stores a subset of data, typically transient in nature, so that future requests for that data are served up faster than is possible by accessing the data’s primary storage location. Caching allows you to efficiently reuse previously retrieved or computed data.

What is Distributed Caching?

Distributed caching is a technique where cache data is spread across multiple servers within a network, enhancing performance and scalability by reducing load on primary data stores. It improves application responsiveness by storing frequently accessed data closer to the application, reducing latency and improving overall system efficiency.

Requirements Gathering for Distributed Cache Design

1. Functional Requirements for Distributed Cache Design

• Read Data: Quickly retrieve data from the cache.
• Write Data: Store data into the cache.
• Eviction Policy: Automatically evict least recently/frequently used items.
• Replication: Replicate data across multiple nodes for fault tolerance.
• Consistency: Ensure data consistency across nodes.
• Node Management: Add and remove cache nodes dynamically.

2. Non-Functional Requirements for Distributed Cache Design

• Performance: Low latency for read and write operations.
• Scalability: System should scale horizontally by adding more nodes.
• Reliability: Ensure high availability and fault tolerance.
• Durability: Persist data if required.
• Security: Secure access to the cache system.

Use Case Diagram for Distributed Cache

A use case diagram helps visualize the interactions between users and the system.

Capacity Estimation for Distributed Cache

Capacity estimation involves calculating the expected load on the system.

1. Traffic Estimate

• Read Traffic: Estimate the number of read requests per second.
• Write Traffic: Estimate the number of write requests per second.

For example, if we expect 10,000 reads per second and 1,000 writes per second, our cache should handle this load.

2. Storage Estimate

• Data Size: Estimate the average size of each cache entry.
• Total Data: Calculate the total amount of data to be stored in the cache.

If each entry is 1KB and we have 1 million entries, the total storage required is 1GB.

3. Bandwidth Estimate

• Read Bandwidth: Calculate the bandwidth needed for read operations.
• Write Bandwidth: Calculate the bandwidth needed for write operations.

For example, if each read operation is 1KB and we have 10,000 reads per second, the read bandwidth is 10MB/s.

4. Memory Estimate

• Node Memory: Estimate the amount of memory required per node.
• Total Memory: Calculate the total memory required across all nodes.

If each node handles 10GB of data and we have 10 nodes, the total memory required is 100GB.

High-Level Design for Distributed Cache Design

The high-level design of a distributed cache system outlines the overall architecture, key components, and their interactions. It focuses on the big picture, ensuring that the system is scalable, fault-tolerant, and efficient. A high-level design outlines the overall architecture of the system.

The high-level design of a distributed cache system, as illustrated in the above diagram, outlines the major components and their interactions to achieve a scalable, fault-tolerant, and efficient caching mechanism. Key components include:

• Clients:
  • The client is any application or service that interacts with the cache system to perform read and write operations. Clients send requests to the cache to retrieve or update data.
• Cache:
  • This is the central component that manages the distributed caching logic. It receives requests from clients and forwards them to the appropriate cache nodes.
  • The cache is responsible for handling read and write operations and ensuring data consistency across the distributed system.
• Nodes (Node 1, Node 2, ..., Node n):
  • These are individual servers or instances that store the cached data. Each node is responsible for a portion of the overall cache and handles read and write requests forwarded by the central cache component.
  • The nodes store data in an in-memory data store like Redis or Memcached, allowing for quick data access and manipulation.
• Data Source:
  • This represents the persistent backend database or data store. If the requested data is not found in the cache (cache miss), the system fetches the data from the data source.
  • The fetched data is then cached for future requests to improve performance.
• Monitoring Service:
  • The monitoring service is responsible for tracking the performance and health of the cache system. It collects metrics such as cache hits, cache misses, and node health status.
  • This information is used to ensure the system operates efficiently and to identify and resolve any issues promptly.

Low-Level Design for Distributed Cache

The low-level design (LLD) of the distributed cache system, as depicted in the below provided diagram, outlines the detailed interactions and responsibilities of each component in the system. This design delves into specific classes or modules, their functions, and how they collaborate to achieve the desired functionality. A low-level design provides detailed descriptions of system components and interactions.

Key Components of the Low-Level Design include:

• CacheClient:
  • Initiates requests for data retrieval or updates.
  • Interacts with the CacheManager to perform operations.
• SystemInitializer:
  • Responsible for setting up and initializing the cache system.
  • Ensures all components are properly configured and ready to handle requests.
• SystemLogger:
  • Logs various activities and events within the system.
  • Useful for debugging and monitoring purposes.
• CacheManager:
  • Central coordinator for cache operations.
  • Receives requests from CacheClient, forwards them to appropriate components, and manages overall cache logic.
• CacheReplicator:
  • Handles data replication to ensure fault tolerance.
  • Ensures that data is consistently replicated across multiple CacheNodes.
• Load Balancer:
  • Distributes incoming requests evenly across multiple CacheServers.
  • Ensures efficient utilization of resources and avoids overloading any single server.
• CacheServer:
  • Manages one or more CachePartitions.
  • Acts as an intermediary between the CacheManager and CachePartitions.
• CachePartition:
  • Subdivision of a CacheServer.
  • Stores a subset of the overall cache data.
  • Ensures data is properly stored and retrieved.
• CacheNode:
  • Actual storage entity within a CachePartition.
  • Stores data in-memory and handles CRUD operations.
• DataStore:
  • Persistent backend database or data source.
  • Provides durable storage and serves as a fallback for cache misses.

Database Design for Distributed Cache

A distributed cache system uses a combination of in-memory storage and persistent backend databases to provide high-speed data access and durability. The database design is a crucial part of this system, ensuring data consistency, fault tolerance, and efficient management of cache entries. While a distributed cache typically doesn't use a traditional database, it can interface with a database for persistence.

Let's see the CacheEntry Table

• Purpose: To store cache data persistently.
• Columns:
  • key: A unique identifier for each cache entry. It ensures that each key in the cache is unique.
  • value: The actual data being cached. This can be any data type, often stored as a text or binary blob.
  • expiration_time: The time at which the cache entry becomes invalid. This helps manage cache eviction.
  • created_at: The timestamp when the cache entry was created. Useful for auditing and cache management.
  • updated_at: The timestamp when the cache entry was last updated. Automatically updated on each modification.

Let's see the SQL of above database table:

CREATE TABLE CacheEntry (
    key VARCHAR(255) PRIMARY KEY,
    value TEXT,
    expiration_time TIMESTAMP,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);

Microservices and APIs Used in Distributed Cache Design

In a distributed cache system, microservices play a crucial role in ensuring modularity, scalability, and maintainability. Each microservice handles a specific set of functionalities and interacts with other microservices through well-defined APIs. Here’s a detailed explanation of the microservices and the APIs involved:

1. Cache Service

The Cache Service is responsible for handling read and write operations on the cache.

1. Set Cache Data API:

Endpoint: POST /cache

Request:

{
  "key": "string",
  "value": "string",
  "ttl": "integer"  // Time-to-live in seconds
}

Response:

{
  "status": "success",
  "message": "Data cached successfully."
}

2. Get Cache Data API:

Endpoint: GET /cache/{key}

Request: None (key is part of the URL)

Response:

{
  "key": "string",
  "value": "string",
  "ttl": "integer"  // Remaining time-to-live in seconds
}

3. Delete Cache Data API:

Endpoint: DELETE /cache/{key}

Request: None (key is part of the URL)

Response:

{
  "status": "success",
  "message": "Data deleted successfully."
}

2. Replication Service

The Replication Service manages the replication of data across multiple cache nodes to ensure data availability and fault tolerance.

1. Replicate Data API:

Endpoint: POST /replicate

Request:

{
  "key": "string",
  "value": "string"
}

Response:

{
  "status": "success",
  "message": "Data replicated successfully."
}

2. Get Replication Status API

Endpoint: GET /replication/status/{key}

Request: None (key is part of the URL)

Response:

{
  "key": "string",
  "replication_status": "string"  // e.g., "completed", "in_progress", "failed"
}

3. Node Management Service

The Node Management Service handles the addition and removal of cache nodes, ensuring the system can scale dynamically.

1. Add Node API

Endpoint: POST /node/add

Request:

{
  "node_id": "string",
  "node_address": "string"
}

Response:

{
  "status": "success",
  "message": "Node added successfully."
}

2. Remove Node API

Endpoint: DELETE /node/remove/{node_id}

Request: None (node_id is part of the URL)

Response:

{
  "status": "success",
  "message": "Node removed successfully."
}

4. Coordinator Service

The Coordinator Service manages consistent hashing, rebalancing, and overall coordination of the cache nodes.

1. Rebalance Data API

Endpoint: POST /rebalance

Request:

{
  "action": "start"  // or "stop" to halt rebalancing
}

Response:

{
  "status": "success",
  "message": "Rebalancing initiated."
}

2. Get Rebalance Status

Endpoint: GET /rebalance/status

Request: None

Response:

{
  "rebalance_status": "string"  // e.g., "in_progress", "completed", "not_started"
}

5. Monitoring and Management Service

This service tracks performance metrics and health status of the cache system, providing insights for administrators.

1. Get Cache Metrics API

Endpoint: GET /metrics

Request: None

Response:

{
  "cache_hits": "integer",
  "cache_misses": "integer",
  "node_health": [
    {
      "node_id": "string",
      "status": "string"  // e.g., "healthy", "unhealthy"
    }
  ]
}

2. Get Node Health

Endpoint: GET /node/health/{node_id}

Request: None (node_id is part of the URL)

Response:

{
  "node_id": "string",
  "status": "string"  // e.g., "healthy", "unhealthy"
}

Scalability for Distributed Cache Design

To ensure scalability, the system should support horizontal scaling, load balancing, and efficient data distribution.

• Horizontal Scaling
  • Add Nodes: Dynamically add more nodes to the cache cluster.
  • Rebalance Data: Use consistent hashing to minimize data movement when nodes are added or removed.
• Load Balancing
  • Distribute Requests: Use a load balancer to distribute client requests evenly across cache nodes.
• Efficient Data Distribution
  • Consistent Hashing: Ensures even distribution of data across nodes and minimizes data rebalancing.

Conclusion

Designing a distributed cache involves addressing various functional and non-functional requirements, estimating capacity, and creating both high-level and low-level designs. Proper planning and implementation of scalability and fault tolerance mechanisms are crucial for building an efficient and reliable distributed cache system.