Chapter 13 Redis In-Memory Database

Objectives

 Upon completion of this course, you will be able to:

 Know Redis application scenarios.
 Learn Redis data types.
 Master Redis optimization methods.
 Understand Redis service development.

1 Huawei Confidential
 Contents

1. Redis Application Scenarios

2. Redis Service Process

3. Redis Features and Data Types

4. Redis Optimization

5. Redis Application Cases

2 Huawei Confidential
Redis Overview
 Redis is a network-based, high-performance key-value in-memory database.
 Redis is similar to Memcached. Besides, it supports data persistence and diverse
data types. It also supports the calculation of the union, intersection, and
complement of sets on the server as well as multiple sorting functions.
 Redis has the following features:
 High performance
 Low latency
 Access to diverse data structures
 Persistence

3 Huawei Confidential
Redis Application Scenarios
 Redis supports flexible data structures and various data operations and is applicable to the
following scenarios:
 Obtaining the latest N pieces of data, for example, the latest articles from a certain website.
 Obtaining the top N applications from a ranking list. This operation is based on a certain condition, for
example, sorting by times that people click "Like", while the preceding operation gives priority to time.
 Applications that require precise expiration time, for example, user session information.
 Counter applications, for example, counters that record website access times.
 Constructing a queue system, for example, a message queue.
 Cache, for example, table data that is frequently accessed in a cached relational database.
 Publish/Subscription (pub/sub).
 SMS verification code (The expire parameter is used to set the expiration time of the verification code).

4 Huawei Confidential
 Contents

1. Redis Application Scenarios

2. Redis Service Process

3. Redis Features and Data Types

4. Redis Optimization

5. Redis Application Cases

5 Huawei Confidential
Redis Architecture
 The Redis architecture is without a central control node. Node status information is exchanged between
nodes by using the Gossip protocol.
 Each node maintains the mapping relationship from a key to a server.
 A client can send a service request to any node, and the node redirects the client instead of forwarding the
request.
 If the cluster topology changes during the duration after the client sends the first request and before the
request is redirected, the second redirection request will be redirected again until a target server is found.

6 Huawei Confidential
Redis Data Reading and Writing Processes (1)

Redis Cluster

1. Access Server1 Server3

Client
2. Cluster
topology
3. Request

Server2

4. Redirect to Server3

5. Request

6. Response

7 Huawei Confidential
Redis Data Reading and Writing Processes (2)
 Redis data reading and writing processes are as follows:
1. The client accesses any server node in the cluster and sends the cluster nodes request.

2. The server node returns the cluster topology, including the cluster node list and the mapping relationship
between slots and nodes. The client caches the cluster topology in memory.

3. The client calculates the slot of the key based on hash (KEY)%16384 and queries the mapping relationship
between slots and nodes, and then accesses the Server2 node that the key belongs to read and write data.

4. Server2 receives the request sent by the client and checks whether the key exists. If the key does not exist,
Server2 informs the client of redirecting the request to the Server3 node. If the key exists, Server2 returns
the service operation results.

5. The client receives the redirection response and sends a reading and writing request to Server3.

6. Server3 receives the request and processes the request the same way as that in step 4.

8 Huawei Confidential
 Contents

1. Redis Application Scenarios

2. Redis Service Process

3. Redis Features and Data Types

4. Redis Optimization

5. Redis Application Cases

9 Huawei Confidential
Redis Feature - Multiple Databases
 Multiple databases
 Each database is named in ascending order starting from 0 and cannot be customized.
 By default, Redis supports 16 databases. Users can change the default number by modifying
the databases parameter.
 By default, Redis uses database 0.
 SELECT number: Switches to a desired database by selecting a number.
 The multiple databases are not completely isolated from each other. For example, the flushall
command takes effect on all databases.
 flushall: Clears data in all databases of a Redis instance.
 flushdb: Clears data in the current database.

10 Huawei Confidential
Basic Commands of Redis
 Obtain the key name that meets rules.
 Expression of keys: (?,* ,[],\?)

 Check whether a key exists.

 exists key

 Delete one or more keys.

 del key
 del key1 key2

 Obtain data type of the obtained key value.

 The return value can be of the string, hash, list, set, or zset type.

 Note: Redis commands are case insensitive.

11 Huawei Confidential
Redis Data Type - String
 The string type is the most basic data type in Redis. It can store content in any
form, including binary data and even an image (binary content). The maximum
capacity for storing a string value is 1 GB.
 Commands
 set/get(setnx)
 mset/mget
 incr/decr/incrby/decrby/incrbyfloat
 append
 strlen

12 Huawei Confidential
Redis Data Type - Hash
 A hash value stores the mapping between fields and field values. The fields and
field values must be strings. A hash key can store a maximum of 232 -1 fields.
 Hash is suitable for storing objects.
 Redis can add or delete fields for any key without affecting other keys.
 Commands:

hset/hget/hmset/hmget/hgetall(hsetnx)
hexists (Check whether the attribute in the key exists.)
hincrby (The hash type does not have the hincr command.)
hdel
hkeys/hvals
hlen (Obtain the number of fields contained in a key.)

13 Huawei Confidential
Redis Data Type - List
 List is an ordered string list. The list is implemented using a bidirectional link
(linked list). List can also be used as a queue.
 A key of the list type can contain a maximum of 232 -1 elements.
 Commands:
lpush/rpush/lpop/rpop;
llen/lrange (-1 indicates the location of the last element.)
lrem (lrem key count value) count has the following three situations:
count > 0: searches from the table header to the table tail and removes the elements whose values are equal to value. The
number of elements is count.
count < 0: searches from the table tail to the table header and removes the elements whose values are equal to value. The
number of elements is the absolute value of count.
count = 0: removes all elements whose values are equal to value from the table.
lindex: queries the data of a specified corner mark.
lset: changes the value of a specified corner mark.
ltrim: truncates and retains the specified data.
linsert: inserts elements before and after a specified element.
rpoplpush: transfers an element from one list to another.

14 Huawei Confidential
Redis Data Type - Set
 The elements in a set are not duplicate and are unordered. A key of the set type
32
can store a maximum of 2 -1 elements.
 Commands

sadd/smembers/srem/sismember;
sdiff (difference set)/sinter (intersection set)/sunion (union set);
sdiffstore/sinterstore/sunionstore;
scard (Obtains the set length.)/spop (Randomly takes an element out of the set and deletes it.);
srandmember key [count]:
If count is a positive number and less than the set cardinality, the command returns an array
containing count different elements.
If count is greater than or equal to the set cardinality, the entire set is returned.
If count is a negative number, the command returns an array. The elements in the array may
appear multiple times, and the length of the array is the absolute value of count.

15 Huawei Confidential
Redis Data Type - Sorted Set
 Each element in the set is associated with a score based on the set type. In this
way, N elements with the highest scores can be easily obtained.
 Commands

zadd/zscore/zrange/zrevrange/
zrangebyscore (A closed interval is used by default. Users can use "(" to adopt an
open interval.)
zincrby/zcard/zcount (Obtains the number of elements in a specified score range.
A closed interval is used by default. Users can use "(" to adopt an open interval.)
zrem/zremrangebyrank/zremrangebyscore (A closed interval is used by default.
Users can use "(" to adopt an open interval.)
Extension: +inf (positive infinity) -inf (negative infinity)

16 Huawei Confidential
Setting TTL of a Key in Redis (Using Expire Command)
 In Redis, users can run the Expire command to set the time to live (TTL) of a key. After the TTL
expires, Redis automatically deletes the key.
 Expire: Sets the TTL (in seconds).
 Pexpire: Sets the TTL (in milliseconds).
 ttl/pttl: Checks the remaining TTL of the key.
 Persist: Cancels the TTL.
 expireat [key]: Indicates the unix timestamp 1351858600.
 pexpireat [key]: Indicates the unix timestamp 1351858700000. (unit: ms)

 Application scenarios:
 Time-limited preferential activity information
 Website data cache (for data that needs to be updated periodically, for example, bonus point rankings)
 Limiting the frequency to access a website (for example, a maximum of 10 times per minute).

19 Huawei Confidential
Redis Pipeline
 The pipeline function of Redis is not available in the command line. However,
Redis supports pipelines and can be used on Java clients (jedis).
 Test results:
 If pipeline is not used, it takes 328 ms to insert 1,000 data records.
for (int i = 0; i < 1000; i++) {
jedis.set("test"+i, "test"+i);
}

 If pipeline is used, it takes 37 ms to insert 1,000 data records.

Pipeline pipelined = jedis.pipelined();

for (int i = 0; i < 1000; i++) {
pipelined.set("test"+i, "test"+i);
}
pipelined.sync();

20 Huawei Confidential
Data Sorting in Redis (Using sort Command)
 The sort command can sort the list, set, and ordered set.
 sort key [desc] [limit offset count]
 by reference key (The reference key can be a character string or a field of the hash type. The format of the hash type is
"key name -> field name".)
 If the reference key does not contain an asterisk (*), the data is not sorted.
 If the reference key of an element does not exist, the value of the reference key is 0 by default.

 Extended get parameter: The rule of the get parameter is the same as that of the by parameter. get #
(Returns the value of the element.)
 Extended store parameter
 Use the store parameter to save the sort result to a specified list.

 Performance optimization:
 Reduce the number of elements in the key to be sorted as much as possible.
 Use the limit parameter to obtain only the required data.
 If there is a large amount of data to be sorted, use the store parameter to cache the result.

21 Huawei Confidential
Redis Task Queues
 Task queue: Use lpush and rpop to implement common task queues.
 Priority queue:
 brpop key1 key2 key3 timeout second

22 Huawei Confidential
Redis Persistence
 Redis supports two persistence modes, which can be used separately or
together.
 RDB mode (Default)
 AOF mode

23 Huawei Confidential
Redis Persistence - RDB
 The persistence in Redis Database (RDB) mode is implemented through snapshots. When certain
conditions are met, Redis automatically takes snapshots of all data in the memory and stores the
data to a disk. By default, the data is stored in the dump.rdb file.
 The time when Redis takes snapshots (in the redis.conf configuration file) is as follows:
 save 900 1: A snapshot is taken if at least one key is changed within 900 seconds.
 save 300 10
 save 60 10000

 Run the save or bgsave command to enable Redis to perform snapshot operations.
 The difference between the two commands is that the save command is used by the main process to
perform snapshot operations, which block other requests, and the bgsave command is used by Redis to
execute the fork function to copy a subprocess for snapshot operations.

24 Huawei Confidential
Redis Persistence - AOF (1)
 Append Only File (AOF) persistence is implemented through log files. By
default, AOF is disabled in Redis. Users can enable it by setting the appendonly
parameter.
 appendonly yes
 Synchronization policies for writing commands of Redis:
 appendfsync always: The command is executed every time.
 appendfsync synchronizedsec: By default, the synchronization is performed every
second (recommended, default).
 appendfsync no: The synchronization is performed by the operating system every 30
seconds.

25 Huawei Confidential
Redis Persistence - AOF (2)
 Dynamically switch the Redis persistence mode from RDB to AOF (Redis 2.2 or later is supported).
 CONFIG SET appendonly yes
 (Optional) CONFIG SET save ""

 Note: When Redis is started, if both RDB persistence and AOF persistence are enabled, the
program preferentially uses the AOF mode to restore the data set because the data stored in the
AOF mode is the most complete. If the AOF file is lost, the database is empty after the startup.
 Note: To switch the running Redis database from RDB to AOF, users can use the dynamic
switchover mode and then modify the configuration file. (Do not modify the configuration file on
your own and restart the database. Otherwise, the data in the database is empty.)

26 Huawei Confidential
Redis Memory Usage
 1 million key-value pairs (key ranges from 0 to 999999, and value is hello
world) on a 32-bit laptop use 100 MB memory.
 If a 64-bit operating system is used, more memory is occupied. This is because
the pointer in the 64-bit operating system occupies eight bytes. However, the
64-bit operating system supports larger memory. It is recommended that the
64-bit server be used to run large-scale Redis services.

27 Huawei Confidential
 Contents

1. Redis Application Scenarios

2. Redis Service Process

3. Redis Features and Data Types

4. Redis Optimization

5. Redis Application Cases

28 Huawei Confidential
Redis Optimization (1)
 Simplify key names and values.
 Key name: The key name should be as simple as possible, but do not use incomprehensible key names just for saving
space.
 Key value: If the number of key values is fixed, the values can be represented by 0 and 1, for example, male/female or
right/wrong.

 If data persistence is not required in service scenarios, disable all data persistence modes to achieve optimal
performance.
 Optimize internal coding (only need to have a basic understanding of it).
 Redis provides two internal coding methods for each data type. Redis can automatically adjust the coding method in
different scenarios.

 SLOWLOG [get/reset/len]
 The commands whose execution time is larger than the value (in microseconds, one second = one million microseconds)
specified by slowlog-log-slower-than will be recorded.
 slowlog-max-len determines the maximum number of logs that can be saved in slowlog.
29 Huawei Confidential
Redis Optimization (2)
 Modify the memory allocation policy of the Linux kernel.
 Add vm.overcommit_memory = 1 to /etc/sysctl.conf and restart the server.
 Alternatively, run the sysctl vm.overcommit_memory=1 command (take effect
immediately).

30 Huawei Confidential
Redis Optimization (3)
 Disable Transparent Huge Pages (THP).
 THP may cause memory locks and affect Redis performance. It is recommended that
this function be disabled.
 THP is used to improve memory management performance.
 THP is not supported in 32-bit RHEL 6.
 Run the following command as user root:
 echo never > /sys/kernel/mm/transparent_hugepage/enabled
 Add the command to the /etc/rc.local file.

31 Huawei Confidential
Redis Optimization (4)
 Modify the maximum number of TCP connections in Linux.
 This parameter determines the length of a completed queue (after three-way
handshake) in the TCP connection. The value must be less than or equal to the
/proc/sys/net/core/somaxconn value defined by the Linux system. The default value
is 511 for Redis and 128 for Linux. When the system has a large number of
concurrent requests and the client responds slowly, users can set the value by
referring to the two parameters.
 echo 511 > /proc/sys/net/core/somaxconn
 Note: This parameter does not limit the maximum number of Redis connections. To
limit the maximum number of Redis connections, modify the maxclients parameter.
The default maximum number of connections is 10000.
32 Huawei Confidential
Redis Optimization (5)
 Limit the Redis memory size.
 Run the info command of Redis to view the memory usage.
 If maxmemory is not set or set to 0, the memory size is not limited in a 64-bit system, and the maximum memory size is
3 GB in a 32-bit system.
 Modify maxmemory and maxmemory-policy in the configuration file.
 maxmemory indicates the maximum memory.
 maxmemory-policy indicates the data clearance policy when the memory is insufficient.

 If the total data volume is not large and the memory is sufficient, users do not need to limit the memory used
by Redis. If the data volume is unpredictable and the memory is limited, limit the memory used by Redis to
prevent Redis from using the swap partition or prevent OOM errors.
 Note: If the memory is not limited, the swap partition is used after the physical memory is used up. In this case, the
performance is low. If the memory is limited, data cannot be added after the specified memory is reached. Otherwise, an
OOM error is reported. Users can set maxmemory-policy to delete data when the memory is insufficient.

33 Huawei Confidential
Redis Optimization (6)
 Redis is a single-thread model. Commands from the client are executed in
sequence. Therefore, users can use pipelines or commands to add multiple
pieces of data at a time, for example:

set mset
get mget
lindex lrange
hset hmset
hget hmget

36 Huawei Confidential
 Contents

1. Redis Application Scenarios

2. Redis Service Process

3. Redis Features and Data Types

4. Redis Optimization

5. Redis Application Cases

37 Huawei Confidential
Application Development Case Analysis
 Business objective
 Some online recommendation services are demanding of velocity because users send
a service request upon obtaining the recommendation result. If the latency is high,
user experience will be affected. Therefore, real-time access to the recommendation
result must be a concern for online recommendation services.
 Service solution
 Redis functions as a cache mechanism. This reduces times and data volume that a
user reads data from a database and a file system.
 The calculated recommendation results are cached to Redis. Next time when users
obtain the same recommendation results, they can directly read data from the cache.

38 Huawei Confidential
Application Development Case Analysis
 Data structure design
 The user information used in the calculation is stored and obtained using the hash structure. The key is
userinfo-<user id>, and the fields contain the user attributes, such as the name, gender, age, and hobby.
 Example: userinfo-19810101,name,zhangsan
 userinfo-19810101,sex,female

 Multiple recommendation results (offerings) may exist. To avoid repeated recommendation of an offering,
the set result is used for storage and access. The key is designed as res-<user id>.

 Data reading and writing principles

 The MapReduce task periodically imports user data from the backend data storage source (Hbase) to Redis every day.
 The service system obtains data from Redis. If Redis data fails to be obtained, the service system then obtains data from
the backend HBase or by real-time computing, and then writes the data simultaneously to Redis.

39 Huawei Confidential
 Quiz

1. Which Redis data structure is appropriate when the top N records for an
application need to be obtained?

2. Does the Redis server forward a key operation request that does not belong to its
node to the correct node?

40 Huawei Confidential
 Summary

 This course introduces the application scenarios, features, data types, and
service data reading and writing processes of the Redis component. After
learning this course, users can select a proper Redis data structure based
on the specific service scenario to access Redis data.

41 Huawei Confidential
 Recommendations

 Huawei Cloud Official Web Link:

 https://www.huaweicloud.com/intl/en-us/
 Huawei MRS Documentation:
 https://www.huaweicloud.com/intl/en-us/product/mrs.html
 Huawei TALENT ONLINE:
 https://e.huawei.com/en/talent/#/

42 Huawei Confidential
Thank you. 把数字世界带入每个人、每个家庭、
每个组织，构建万物互联的智能世界。
Bring digital to every person, home, and
organization for a fully connected,
intelligent world.

Copyright© 2020 Huawei Technologies Co., Ltd.

All Rights Reserved.

The information in this document may contain predictive

statements including, without limitation, statements regarding
the future financial and operating results, future product
portfolio, new technology, etc. There are a number of factors that
could cause actual results and developments to differ materially
from those expressed or implied in the predictive statements.
Therefore, such information is provided for reference purpose
only and constitutes neither an offer nor an acceptance. Huawei
may change the information at any time without notice.