A while ago, Tencent Cloud was engaged in activities. Ha C, I bought a lightweight server and deployed my own website.
Everything is proceeding in an orderly manner.
Until one morning, I opened my website as always, and found that the website could not be opened.
So I conducted a series of investigations.
1. Check the log
The first thing that comes to mind is to log in to the server and check the logs of abnormal logins.
Boy, I found out that the server can't log in!
1) VNC login server
The first thing that comes to mind is that password login is disabled.
So I used VNC to log in at the backend of Tencent Cloud.
If you can't log in remotely via client SSH, you can log in to the server via VNC login.
2) View the sshd_config file
After checking the
/etc/ssh/sshd_configfile, I found that it was indeed modified:
PasswordAuthentication no # means that password login is not allowed
If password login is disabled, it should be used to log in with the private key.
Change to yes first, then restart sshd
systemctl restart sshd
3) Use the terminal to log in again
After the modification, use the terminal tool locally to log in again, because the VNC login tool is too difficult to use.
Found that you can log in.
Good guy, don't talk about yards!
Added a key pair to my server:
4) View the login log
- Use the last and history commands to check the login log and operation log
last #View all logged in ip
history #View the command record of the operation
It is not surprising that there is no abnormal IP. If it is logged in, the log log is likely to be deleted.
- Use the lastb command to check again:
lastb #Used to list information about users who failed to log in to the system
lastbInterpretation of results:
第一列：用户名 第二列：终端位置 第三列：登录ip或者内核 第四列：开始时间 第五列：结束时间（still login in 还未退出 down 直到正常关机 crash 直到强制关机） 第六列：持续时间
The above results indicate that the server was violently hit against the database .
The IP should be through the proxy. In the second picture, the other party directly used root as the user name and kept hitting the database. It seems that the user name was found correctly. In the end, I really logged in and modified my secret key pair.
Check the IP:
The IP is a foreign country, it is difficult to find the location, and it may be a proxy IP.
2. Find the Trojan file
1) Use the top command to take a look
topcommands can't display the Trojan horse process at all, it looks like a normal look, because the
topcommand may have been modified by the intruder:
2) busybox command
busybox topcan see the hidden CPU-intensive process, the original
tophas been modified virus process can not be displayed, you must
Download the troubleshooting tool given by Tencent Cloud
[[email protected] ~]# wget https://tao-1257166515.cos.ap-chengdu.myqcloud.com/busybox --2020-12-14 15:12:59-- https://tao-1257166515.cos.ap-chengdu.myqcloud.com/busybox Resolving tao-1257166515.cos.ap-chengdu.myqcloud.com (tao-1257166515.cos.ap-chengdu.myqcloud.com)... 18.104.22.168, 22.214.171.124, 126.96.36.199, ... Connecting to tao-1257166515.cos.ap-chengdu.myqcloud.com (tao-1257166515.cos.ap-chengdu.myqcloud.com)|188.8.131.52|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1001112 (978K) [application/octet-stream] Saving to: ‘busybox.1’ 100%[======================================>] 1,001,112 1.36MB/s in 0.7s [[email protected] ~]# cp busybox /usr/bin/ [[email protected] ~]# busybox top -bash: /usr/bin/busybox: Permission denied [[email protected] ~]# cd /usr/bin/ [[email protected] bin]# chmod 777 /usr/bin/busybox [[email protected] ~]# busybox top
Caught the Trojan horse file:
The above sees that the CPU occupancy rate has reached nearly 100%, and mining is no doubt.
Finally, after investigating with Tencent Cloud technology for a long time, the following Trojan files and directories were finally found:
/tmp/.X25-unix/.rsync/c/tsm64 /tmp/.X25-unix/.rsync/c/tsm32 /tmp/.X25-unix/.rsync/a/kswapd0 /usr/bin/systemd-network /usr/bin/kswaped
Finally, lock the name of the mining process as pamdicks , then kill the Trojan horse process, and then delete the Trojan horse file, it should be fine .
If you do not enter the full name, the
ls、ll、lsattrfile viewing command will not display the Trojan file at all:
Before deleting, see what the mining process is:
ls -lh /proc/5445/fd
top -H -p 5445
This pamdicks process has 6 child threads:
Finally, it was traced to a binary file, which touched the scope of knowledge, and it could not be opened. Just delete it.
3. Delete the Trojan file
1) Modify authorized_keys
authorized_keysDelete the public key of the file first . When I perform
rmwhen the command, the intruder put my
authorized_keyspapers add a
+ilock can not be deleted, so sad:
chattr +i /etc/authorized_keys
means that the file cannot be deleted, cannot be changed, or moved
I really don't talk about code, and I
chattrdeleted the commands of my server . The server was hacked. It
chattris a common operation to delete commands .
Sure enough, the chattr command is not found:
You can only
chattrinstall it back manually , the
yum install e2fsprogs
[[email protected] run]# which chattr /usr/bin/chattr
[[email protected] .ssh]# chattr -i authorized_keys [[email protected] .ssh]# echo > authorized_keys [[email protected] .ssh]# cat authorized_keys
2) Execute the rm command to delete the Trojan file
Kill and delete the found Trojan files:
[[email protected] .ssh]# kill -9 5445 [[email protected] .ssh]# chattr -i /usr/bin/pamdicks [[email protected] .ssh]# rm /usr/bin/pamdicks rm: remove regular file ‘/usr/bin/pamdicks’? y [[email protected] .ssh]# rm /tmp/.X25-unix/.rsync/c/lib/64/tsm rm: remove regular file ‘/tmp/.X25-unix/.rsync/c/lib/64/tsm’? y
After the deletion, the CPU usage drops:
After the Trojan file is cleaned up, finally disable password login on the server, and log in with the generated secret key pair instead.
Temporarily come to an end.
4. Find the source of the attack-Redis
A few days later, when I opened Redis, I found that Redis had strange key values.
I didn't notice this problem before.
Careless! Redis opened the port and did not set a password.
Although I don't understand how this bunch of things are injected into my server. But the
crackitkey is very strange.
The following information was found online:
Redis Crackit vulnerability: A
hacker remotely accesses the redis service, clears the redis database, writes his own ssh login public key, and then backs up the redis database as /root/.ssh/authotrized_keys.
This successfully writes your own public key to the authenticated_keys of ssh, and directly logs in to the hacked host as root without a password.
That is to say, it is very likely that my server was not logged in by brute force hitting the database, but was attacked using Redis as an entry point.
Use the top command to take a look, I'm going! The Trojan file is up again. It seems that if the Redis key value is not cleaned up, the Trojan file will continue to be downloaded and executed:
This kswapd0 is a bit familiar.
The high occupancy of kswapd0 is due to insufficient physical memory. The swap partition and memory paging operation are used to exchange data, which leads to high CPU occupancy.
However , this
kswapd0is a blind trick, and the command behind it is to execute the Trojan horse file.
So this Redis has not been resolved, the intruder will continue to use your Redis loopholes to continue to invade your server next time.
The true sense of
kswapd0the process is this:
In order to reproduce this process, I executed the Redis value command again:
[[email protected] ~]# ping d.powerofwish.com PING d.powerofwish.com (184.108.40.206) 56(84) bytes of data.
I downloaded a
pm.shscript, open the script:
You can see that this sh script is essentially downloading a png file and giving it executable permissions.
I also directly downloaded the png file, granted permissions, and executed it.
I saw a camouflaged bin script, written after the first delete
And then continue to brush ~ during the
chattrcommand is deleted, the
authorized_keysfile is modified
Finally, it should be the implementation of
/usr/bin/kswapedthis script, start mining.
Then use top to check the CPU, and it instantly rises to close to 100%
Take a look at the package:
tcpdump -i eth0 '((not port 45695) and (not host 127.0.0.1) and (not host 220.127.116.11))'
The one who found this
18.104.22.168is a CDN node in the United States, follow the vine and export the captured package to WirteShark to see:
The red part is the source IP, and the other party's operation to request the database (port 3306).
It is also possible that my server is just a DDoS attack node. Therefore, in order to maintain network security, the Trojan horse files must be processed in a timely manner.
Distributed Denial of Service attack (English means Distributed Denial of Service, DDoS for short) refers to multiple attackers in different locations simultaneously launching attacks on one or several targets, or one attacker controls multiple machines located in different locations And use these machines to attack the victims at the same time. Since the attack points are distributed in different places, this type of attack is called a distributed denial of service attack, in which there can be multiple attackers.
The main reason for the server being attacked is that there are too many public network ports exposed, especially Redis6789, MySQL3306, and the server password is too simple.
By modifying the
- Turn off password login, only allow secret key pair login
- Change the default port of ssh to prevent the brute force library from being cracked
- Disable root account direct login, open specific IP access
#只允许用户、IP访问 AllowUsers aliyun [email protected],[email protected]* # 拒绝 zhangsan、aliyun 帐户通过 SSH 登录系统 DenyUsers zhangsan aliyun #使用秘钥登录 AuthorizedKeysFile .ssh/authorized_keys PubkeyAuthentication yes RSAAuthentication yes #禁用密码登录 PasswordAuthentication no
- Redis only allow local access, modify the default port, not exposed to all IP, Redis default
bind 127.0.0.1for a reason
- MySQL only opens access to the required IP
- Set firewall access rules for ports
If you want to be more secure, you can use springboard machines and bastion machines to access.
Back up data regularly, and back up snapshots regularly.
The above is the investigation process and some thoughts of a server being hacked.
If you want to learn about network detection and security audit of a website, you can learn about nmap: https://nmap.org/man/zh/index.html