It’s all started with Redis. So let’s discuss about redis security first.
Redis is a very popular in-memory data structure has wide variety of use case including caching, database & message broker. Redis designed to be run on secure environment and doesn’t encrypt data for sake of performance. Thus become lucrative prey for intruders.
Bombarded with complains, from version 3.2 redis out of the box run in ‘protected mode’ means only listen to loopback interface (127.0.0.1 & ::1) and doesn’t listen on tcp socket (port 0). Thus only resident services of the host have access to redis.
However one of the key application of redis is in distributed systems where redis need to be accessible over network. That was the case for me. Was migring standalone service to highly available load balanced architecture where redis was part of the package. Modified the redis configuration to listen on tcp socket & public IP, skipped the host firewall for later. Perfect recipe !
Next day at QT noticed frontend application constantly complaining about redis. Went to investigate the redis log found some anomalies. someone from Eastern Europe established master slave replication. Loaded data on memory but failed to write on disk.
* Before turning into a slave, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer. * SLAVE OF "C&C IP":8890 enabled (user request from 'id=Numeric-ID addr="C&C IP":35800 fd=9 name= age=1 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32780 obl=0 oll=0 omem=0 events=r cmd=slaveof') * Connecting to MASTER "C&C IP":8890 * MASTER <-> SLAVE sync started * Non blocking connect for SYNC fired the event. * Master replied to PING, replication can continue… * Trying a partial resynchronization (request HASH ID:1). * Full resync from master: ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ:1 * Discarding previously cached master state. * MASTER <-> SLAVE sync: receiving 55664 bytes from master * MASTER <-> SLAVE sync: Flushing old data * MASTER <-> SLAVE sync: Loading DB in memory # Wrong signature trying to load DB from file # Failed trying to load the MASTER synchronization DB from disk
And at second attempt was successful to write at disk
* Connecting to MASTER "C&C IP":8890 * MASTER <-> SLAVE sync started * Module 'system' loaded from ./red2.so * Non blocking connect for SYNC fired the event. # Setting secondary replication ID to HASH-ID, valid up to offset: 1. New replication ID is HASH-ID * MASTER MODE enabled (user request from 'id=Numeric-ID addr="C&C IP":35800 fd=9 name= age=9 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32780 obl=0 oll=0 omem=0 events=r cmd=slaveof') * Module system unloaded * 1 changes in 900 seconds. Saving… * Background saving started by pid 31570 * DB saved on disk
Second anomaly was redis wants to access cron directory in the name of backup
# Failed opening the RDB file root (in server root dir /var/spool/cron) for saving: Read-only file system # Failed opening the RDB file root (in server root dir /etc/cron.d) for saving: Read-only file system # Failed opening the RDB file redis (in server root dir /var/spool/cron) for saving: Read-only file system # Failed opening the RDB file redis (in server root dir /etc/cron.d) for saving: Read-only file system # Failed opening the RDB file tomcat (in server root dir /var/spool/cron) for saving: Read-only file system # Failed opening the RDB file tomcat (in server root dir /etc/cron.d) for saving: Read-only file system # Failed opening the RDB file www-data (in server root dir /var/spool/cron) for saving: Read-only file system # Failed opening the RDB file www-data (in server root dir /etc/cron.d) for saving: Read-only file system # Failed opening the RDB file apache (in server root dir /var/spool/cron) for saving: Read-only file system # Failed opening the RDB file apache (in server root dir /etc/cron.d) for saving: Read-only file system
This cycle continued with dictionary usernames. Having enough status report moved to enable firewall and blocked internet access to redis along with other services
sudo ufw allow from 172.16.0.0/28 to any port 6379/tcp
That’s blocking all incoming request to redis from Internet. Now coming to permission errors. First tried to flush redis db
127.0.0.1:6379[1]> FLUSHALL (error) MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error. # Failed opening the RDB file nginx (in server root dir /var/spool/cron) for saving: Read-only file system # Background saving error * 1 changes in 900 seconds. Saving… * Background saving started by pid 17564
Stopped with same error. Apparently redis db location was changed to /var/spool/cron in memory.
root@bc:~# echo "CONFIG GET *" | redis-cli | grep -e "dir" -e "dbfilename" -A1
dbfilename
nginx
--
dir
/var/spool/cron
created a new directory /etc/redis/redis-db and gave ownership to redis. After that proceed on changing on directory configuration.
127.0.0.1:6379[1]> CONFIG SET dir /etc/redis/redis-db
OK
127.0.0.1:6379[1]> BGSAVE
Background saving started
Had to kill the redis process and restart the service then both frontend & backend started behaving normally. Real life saver turned out to be the fact that redis was running by non privileged user without shell access. That’s why those permission log generated.
root@bc:~# ps -aux | grep redis
root 2482 0.0 0.0 13136 1028 pts/3 S+ 22:08 0:00 grep --color=auto redis
redis 31248 0.1 0.0 50152 3536 ? Ssl 12:18 0:54 /usr/bin/redis-server 127.0.0.1:6379
root@bc:~# cat /etc/passwd | grep redis
redis:x:113:116::/var/lib/redis:/usr/sbin/nologin
