How Redis manages it storage in a distributed concept and how it handles the failover and utilize the performance

Redis is stands for Remote Directory Service. Redis is an open source in memory data structure project implementing a distributed, in-memory key-value database with optional durability. Redis supports different kinds of abstract data structures, such as strings, lists, maps, sets, sorted sets, hyperloglogs, bitmaps, streams and spatial indexes. The Redis project is mainly developed by Salvatore Sanfilippo and is currently sponsored by Redis Labs. Redis was initially released in 10th of May 2009.

Official Redis Logo

Anyone can run atomic operations on these types, like appending to a string; increment the value in a hash; pushing an element to a list; computing set intersection, union and difference; or getting the member with highest ranking in a sorted set.

Redis supports trivial-to-setup master-slave asynchronous replication, with very fast non-blocking first synchronization, auto re-connection with partial re-synchronization on net split. Other than that it includes other features like transactions, Pub/Sub, Lua Scripting, Keys with a limited time to live, LRU eviction of keys and automatic failover. In this blog article, it enplanes how it handles the failover and utilize the performance in Redis.

Redis follows very amazing concept called Redis Virtual Memory. It explains that instead of using Operating System supplied Virtual Memory facility, it uses their own virtual system called, Redis Virtual Memory.

Redis presents its' own clustering mechanism with the Redis release version 3.0 and it provide consistent and resilient data service in more advanced way. In Redis cluster it can scale the storage of data by nodes and it performs by all instances working together.

According to the clustering mechanism of the Redis, it follows  a master/slave setup to improve the availability of the setup, if it makes a failure. In Redis case, it provides away to run Redis installation where data is automatically shared across multiple Redis nodes.

Main things to obtain from Redis is:

1. The ability to automatically split the data set among multiple nodes.
2. The ability to continue operations when a subset of the nodes are experiencing failures are unable to communicate with the rest of the cluster.     

Redis Cluster TCP ports

Another main thing in the Redis cluster is, it requires two TCP connections open. In normally Redis TCP port uses to serve clients, as an example let's take 6379, 16379 will be the data port by adding 10000 to 6379. this second high port is used for cluster bus and it is a node-to-node communication channel using a binary protocol. This cluster bus is used by nodes for failure detection, configuration update, failover authorization and so forth. Clients should never try to communicate with the cluster bus port, but always with the normal Redis command port, however make sure you open both ports in your firewall, otherwise Redis cluster nodes will be not able to communicate.

The command port and cluster bus port offset is fixed and is always 10000.

Note that for a Redis Cluster to work properly you need, for each node:

1. The normal client communication port (usually 6379) used to communicate with clients to be open to all the clients that need to reach the cluster, plus all the other cluster nodes (that use the client port for keys migrations).
2. The cluster bus port (the client port + 10000) must be reachable from all the other cluster nodes.

If you don't open both TCP ports, your cluster will not work as expected.

The cluster bus uses a different, binary protocol, for node to node data exchange, which is more suited to exchange information between nodes using little bandwidth and processing time.


Failover mechanism in Redis

Suppose that it contains three machines, 3 Redis master nodes on separate machines and 3 Redis slaves as one slave per master. In this case it presents distributed slot map approach in an advanced way. As previous mentioned way, it create a connection with each master with respective slave in other machine. It illustrates in below figure.

Master-slave illustration 01

Suppose that one of the server is going down, then Redis will perform automatic failover mechanism and then master of that server handover its' job to the respective slave in another server instead of failed master. Then newly assigned slave becomes as the new acting master.

Assume that first server in the figure is going down and then master A is going down. According to the internal mechanism of the Redis, it handover the duties of master A to the slave A. Then first server is not available furthermore. Then slave A becomes to master A. After that it does not have any slave A for that moment.

Master-server illustration 02

Within couple of minutes, first server starts to work again. Then initial master A, becomes to slave A and initial slave A becomes to master A. Other master-slave pairs does not change.

Master-slave illustration 03

It needs to be  aware with making/creating two masters in one server. Since above failover, there it makes two masters in third server for a small time. However at the end of failover it needs to rearrange the  cluster in a correct way by the administrators.

Redis installation on Docker

To see the performance of the Redis, it can be installed on Docker. For that it can be used Redis images in Docker. It can be used both command line or Docker web interface for installation.

Command line method:

      Step 1: Start with Redis instance

                  $ docker run --name some-redis -d redis

      Step 2: Start with persistent storage

                  $ docker run --name some-redis -d redis redis-server --appendonly yes

              If persistence is enabled, data is stored in the VOLUME /data, which can be used with --volumes-from some-volume-container or -v /docker/host/dir:/data

      Step 3: Connecting via redis-cli

                 $ docker run -it --network some-network --rm redis redis-cli -h some-redis

      Step 4: Other than if you need to use own redis.conf

              You can create your own Dockerfile that adds a redis.conf from the context into /data/, like so.

                FROM redis
                COPY redis.conf /usr/local/etc/redis/redis.conf
                CMD [ "redis-server", "/usr/local/etc/redis/redis.conf" ]

               Alternatively, you can specify something along the same lines with docker run options.

                 $ docker run -v /myredis/conf/redis.conf:/usr/local/etc/redis/redis.conf --name myredis redis redis-server /usr/local/etc/redis/redis.conf

             Where /myredis/conf/ is a local directory containing the redis.conf  file. Using this method means that there is no need for you to have a Dockerfile for Redis cluster.  

Docker User Interface Method:

       Step 1: It can be install a container by using Redis image from App Templates of Docker web interface. Select the Redis image here.

Redis Image on Docker

        Step 2: After selected Redis image, it needs to give a suitable name for the Redis container. It depends on yourself. After giving a name, click on Deploy the container button.

Creating Redis Image

       Step 3 : After creating the Redis container, you will be able to see Redis container on container list. By clicking on Redis container name,you will be able to get the Redis-cli for the any configurations.

Container List

Reference:

[1] Docker Inc. (2019) redis - Docker Hub, Available at: https://hub.docker.com/_/redis (Accessed: ).

[2] redis (2019) Redis cluster tutorial - Redis, Available at: https: https://redis.io/topics/cluster-tutorial  (Accessed: )

[3] Redis (2019) Concepts and Architecture | Redis Labs Documentation Centre, Available at: https://docs.redislabs.com/latest/rs/concepts/ (Accessed: )