How to Build a Redis Cluster within Kubernetes cluster
Introduction:
This KB deals with configuring a Self Hosted redis cluster inside your active Kubernetes cluster. Most of the customers might be using managed redis service, but there we have to compromise on customization and the versions. Also, managed services are more caged. Self-Hosted redis clusters give you freedom and more customization options. Here we are going to discuss how we can configure a self hosted redis cluster. The environment we are implementing this in an active kubernetes cluster.
Okay!! Now we can go ahead and deep dive into Redis Cluster !!!
How to Build a Redis Cluster within Kubernetes cluster
Redis Cluster: provides a path to run a Redis installation where data is automatically sharded across multiple Redis nodes. Redis Cluster provides some ability to continue the operations when some of the nodes fail or are not able to communicate.
Attracting features of Redis cluster
- It has the ability to automatically split your dataset among multiple nodes.
- It has the ability to continue operations when a subset of the nodes are experiencing failures or are unable to communicate with the rest of the cluster.
Every Redis Cluster node requires two TCP connections to open the normal Redis TCP port used to serve clients (Port 6379) and Cluster bus port (Port 16379). Cluster bus is a node-to-node communication channel using a binary protocol. The Cluster bus is used by nodes for failure detection, configuration update and so forth.
Before we dive in too deep, we have to know some key concepts here.
- What normally Redis do ?
Redis is an in-memory database but it is persistent on disk databases, so it provides very high write and read speed is achieved with the limitation of data sets that can’t be larger than memory. Redis is very fast than any other key-value store mechanism.
- What is a key anyway in Redis?
The original intention of Redis was to have a particular key, or identifier, for each individual piece of data. In Redis a single key could refer to multiple (even millions of) pieces of data.The idea of a key was stretched even further because a single piece of data could now span multiple keys.
- What are Shards in redis ?
A shard is a collection of one or more nodes in a Redis cluster. It is created to support replication of data into various nodes in the Redis cluster so that cache remains reachable in case of loss of few nodes. There will be 16384 hash slots in Redis Cluster, and to compute what is the hash slot of a given key, we take the CRC16 of the key modulo 16384. For a better understanding, please verify the below image.
How we build a Redis cluster within a KOPS cluster
- Installation:
The Redis instances are set up as a statefulset within the Kubernetes cluster. Then the redis cluster can be set using the steps below:
- Clone the below repo to get the required files:
$ git clone https://gitlab.com/activelobby/devops-group/prod/tree/develop/redis-cluster
$ kubectl create -f namespace.yaml
$ kubectl create -f storageclass.yaml
$ kubectl create -f redis-cluster.yml
- Cluster scale up:
$ kubectl scale statefulset redis-cluster –replicas=8
- Set one node as master:
$ kubectl exec -n redis redis-cluster-0 — redis-cli –cluster add-node $(kubectl get pod -n redis redis-cluster-6 -o jsonpath='{.status.podIP}’):6379 $(kubectl get pod -n redis redis-cluster-0 -o jsonpath='{.status.podIP}’):6379
- Set other node as slave:
The second new node will join the cluster as a slave and it will automatically bind to the master with the least slaves (in this case, redis-cluster-6)
$ kubectl exec redis-cluster-0 -n redis — redis-cli –cluster add-node –cluster-slave $(kubectl get pod -n redis redis-cluster-7 -o jsonpath='{.status.podIP}’):6379 $(kubectl get pod -n redis redis-cluster-0 -o jsonpath='{.status.podIP}’):6379
- Automatically rebalance the masters:
$ kubectl exec redis-cluster-0 -n redis — redis-cli –cluster rebalance –cluster-use-empty-masters $(kubectl get pod -n redis redis-cluster-0 -o jsonpath='{.status.podIP}’):6379
- Cluster Scale down:
- Remove slave from cluster
$ kubectl exec redis-cluster-7 -n redis — redis-cli cluster nodes | grep myself
ee841fc7357e0e6ff9f145fd19abe4e469f12b70 100.96.19.195:6379@16379 myself,slave 6918e4e91739cd8a1282af9f43fd35371caae64d 0 1541159037000 6 connected
$ kubectl exec redis-cluster-0 -n redis — redis-cli –cluster del-node $(kubectl get pod -n redis redis-cluster-0 -o jsonpath='{.status.podIP}’):6379 ee841fc7357e0e6ff9f145fd19abe4e469f12b70
- Remove master from cluster:
Get ID of master to be removed:
$ kubectl exec redis-cluster-7 -n redis — redis-cli cluster nodes | grep myself
6918e4e91739cd8a1282af9f43fd35371caae64d 100.96.20.159:6379@16379 myself,master – 0 1541159211000 3 connected 10923-16383
Get ID of remaining masters:
$ kubectl exec redis-cluster-7 — redis-cli cluster nodes | grep master | grep -v myself
cb620c589c8ce84ea834f7bfd7b38ab25f9b109b 100.96.16.227:6379@16379 master – 0 1541159267000 2 connected 5461-10922
4c81091b599321e5f5e96cee84f079cfd23b1bf5 100.96.21.206:6379@16379 master – 0 1541159268000 1 connected 0-5460
4c81091b599321e5f5e96cee84f079cfd23b1bf5 100.96.21.206:6379@16379 master – 0 1541159268000 1 connected 0-5460
Move all slots from redis-cluster-6 using reshard command:
$ kubectl exec redis-cluster-0 -n redis — redis-cli –cluster reshard –cluster-yes –cluster-from 6918e4e91739cd8a1282af9f43fd35371caae64d –cluster-to cb620c589c8ce84ea834f7bfd7b38ab25f9b109b –cluster-slots 16384 $(kubectl get pod -n redis redis-cluster-0 -o jsonpath='{.status.podIP}’):6379
After resharding, delete the redis-cluster-6 master node:
$ kubectl exec redis-cluster-0 -n redis — redis-cli –cluster del-node $(kubectl get pod redis-cluster-0 -o jsonpath='{.status.podIP}’):6379 6918e4e91739cd8a1282af9f43fd35371caae64d
Now rebalance the remaining masters to evenly distribute slots:
$ kubectl exec redis-cluster-0 -n redis — redis-cli –cluster rebalance –cluster-use-empty-masters $(kubectl get pod -n redis redis-cluster-0 -o jsonpath='{.status.podIP}’):6379
Now delete the scale down the cluster:
$ kubectl scale statefulset -n redis redis-cluster –replicas=6
Limitations of self hosted Redis cluster
- Self-hosted redis will not auto scale.
- Difficulties in maintenance.
Thanks for your time reading this. Please proceed with creating your 1st Redis Cluster ….
