[Teach you to build a server series] (7) The whole process of troubleshooting a server being hacked

A while ago, Tencent Cloud was engaged in activities. Ha C, I bought a lightweight server and deployed my own website.

Everything is proceeding in an orderly manner.

Until one morning, I opened my website as always, and found that the website could not be opened.

So I conducted a series of investigations.

1. Check the log

The first thing that comes to mind is to log in to the server and check the logs of abnormal logins.

Boy, I found out that the server can't log in!

1) VNC login server

The first thing that comes to mind is that password login is disabled.

So I used VNC to log in at the backend of Tencent Cloud.

If you can't log in remotely via client SSH, you can log in to the server via VNC login.

2) View the sshd_config file

After checking the /etc/ssh/sshd_configfile, I found that it was indeed modified:

PasswordAuthentication no # means that password login is not allowed

If password login is disabled, it should be used to log in with the private key.

Change to yes first, then restart sshd

systemctl restart sshd

3) Use the terminal to log in again

After the modification, use the terminal tool locally to log in again, because the VNC login tool is too difficult to use.

Found that you can log in.

View authorized_keysfile

 vi /root/.ssh/authorized_keys

Good guy, don't talk about yards!

Added a key pair to my server:

4) View the login log

  • Use the last and history commands to check the login log and operation log
last #View all logged in ip
history #View the command record of the operation

It is not surprising that there is no abnormal IP. If it is logged in, the log log is likely to be deleted.

  • Use the lastb command to check again:
lastb #Used to list information about users who failed to log in to the system
Figure one
Figure II

lastbInterpretation of results:

第五列:结束时间(still login in 还未退出  down 直到正常关机 crash 直到强制关机)

The above results indicate that the server was violently hit against the database .

The IP should be through the proxy. In the second picture, the other party directly used root as the user name and kept hitting the database. It seems that the user name was found correctly. In the end, I really logged in and modified my secret key pair.

Check the IP:

The IP is a foreign country, it is difficult to find the location, and it may be a proxy IP.

2. Find the Trojan file

1) Use the top command to take a look

Ordinary topcommands can't display the Trojan horse process at all, it looks like a normal look, because the topcommand may have been modified by the intruder:

Ordinary top command

2) busybox command

Run busybox topcan see the hidden CPU-intensive process, the original tophas been modified virus process can not be displayed, you must busyboxperform the

Download the troubleshooting tool given by Tencent Cloud busybox,

[[email protected] ~]# wget https://tao-1257166515.cos.ap-chengdu.myqcloud.com/busybox
--2020-12-14 15:12:59--  https://tao-1257166515.cos.ap-chengdu.myqcloud.com/busybox
Resolving tao-1257166515.cos.ap-chengdu.myqcloud.com (tao-1257166515.cos.ap-chengdu.myqcloud.com)...,,, ...
Connecting to tao-1257166515.cos.ap-chengdu.myqcloud.com (tao-1257166515.cos.ap-chengdu.myqcloud.com)||:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1001112 (978K) [application/octet-stream]
Saving to: ‘busybox.1’

100%[======================================>] 1,001,112   1.36MB/s   in 0.7s
[[email protected] ~]# cp busybox /usr/bin/
[[email protected] ~]# busybox top
-bash: /usr/bin/busybox: Permission denied
[[email protected] ~]# cd /usr/bin/
[[email protected] bin]# chmod 777 /usr/bin/busybox
[[email protected] ~]# busybox top

Caught the Trojan horse file:

busybox top command

The above sees that the CPU occupancy rate has reached nearly 100%, and mining is no doubt.

Finally, after investigating with Tencent Cloud technology for a long time, the following Trojan files and directories were finally found:


Finally, lock the name of the mining process as pamdicks , then kill the Trojan horse process, and then delete the Trojan horse file, it should be fine .

If you do not enter the full name, the ls、ll、lsattrfile viewing command will not display the Trojan file at all:

Trojan file cannot be displayed, full name is required

Before deleting, see what the mining process is:

ls -lh /proc/5445/fd
 top -H -p 5445

This pamdicks process has 6 child threads:

Finally, it was traced to a binary file, which touched the scope of knowledge, and it could not be opened. Just delete it.

3. Delete the Trojan file

1) Modify authorized_keys

authorized_keysDelete the public key of the file first . When I perform rmwhen the command, the intruder put my authorized_keyspapers add a +ilock can not be deleted, so sad:

chattr +i /etc/authorized_keys
means that the file cannot be deleted, cannot be changed, or moved

I really don't talk about code, and I chattrdeleted the commands of my server . The server was hacked. It chattris a common operation to delete commands .

Sure enough, the chattr command is not found:

You can only chattrinstall it back manually , the centosinstallation process:

yum install e2fsprogs

Successful installation:

[[email protected] run]# which chattr

Empty the authorized_keysfile:

[[email protected] .ssh]# chattr -i authorized_keys
[[email protected] .ssh]# echo > authorized_keys
[[email protected] .ssh]# cat authorized_keys

2) Execute the rm command to delete the Trojan file

Kill and delete the found Trojan files:

[[email protected] .ssh]# kill -9 5445
[[email protected] .ssh]# chattr -i /usr/bin/pamdicks
[[email protected] .ssh]# rm /usr/bin/pamdicks
rm: remove regular file ‘/usr/bin/pamdicks’? y
[[email protected] .ssh]# rm /tmp/.X25-unix/.rsync/c/lib/64/tsm
rm: remove regular file ‘/tmp/.X25-unix/.rsync/c/lib/64/tsm’? y

After the deletion, the CPU usage drops:

After the Trojan file is cleaned up, finally disable password login on the server, and log in with the generated secret key pair instead.

Temporarily come to an end.

4. Find the source of the attack-Redis

A few days later, when I opened Redis, I found that Redis had strange key values.

I didn't notice this problem before.

Redis is hacked

Careless! Redis opened the port and did not set a password.

Although I don't understand how this bunch of things are injected into my server. But the crackitkey is very strange.

The following information was found online:

Redis Crackit vulnerability: A
hacker remotely accesses the redis service, clears the redis database, writes his own ssh login public key, and then backs up the redis database as /root/.ssh/authotrized_keys.
This successfully writes your own public key to the authenticated_keys of ssh, and directly logs in to the hacked host as root without a password.

That is to say, it is very likely that my server was not logged in by brute force hitting the database, but was attacked using Redis as an entry point.

Use the top command to take a look, I'm going! The Trojan file is up again. It seems that if the Redis key value is not cleaned up, the Trojan file will continue to be downloaded and executed:

kswapd0 disguised Trojan horse

This kswapd0 is a bit familiar.

The high occupancy of kswapd0 is due to insufficient physical memory. The swap partition and memory paging operation are used to exchange data, which leads to high CPU occupancy.

However , this kswapd0is a blind trick, and the command behind it is to execute the Trojan horse file./tmp/.x25-unix/.rsynckswapd0

So this Redis has not been resolved, the intruder will continue to use your Redis loopholes to continue to invade your server next time.

The true sense of kswapd0the process is this:

This is normal kswapd0

In order to reproduce this process, I executed the Redis value command again:

[[email protected] ~]# ping d.powerofwish.com
PING d.powerofwish.com ( 56(84) bytes of data.

I downloaded a pm.shscript, open the script:

You can see that this sh script is essentially downloading a png file and giving it executable permissions.

I also directly downloaded the png file, granted permissions, and executed it. ./png

I saw a camouflaged bin script, written after the first delete /usr/binthe directory.

And then continue to brush ~ during the chattrcommand is deleted, the authorized_keysfile is modified

Finally, it should be the implementation of /usr/bin/kswapedthis script, start mining.

Then use top to check the CPU, and it instantly rises to close to 100%

Take a look at the package:

tcpdump -i eth0 '((not port 45695) and (not host and  (not host'

The one who found this a CDN node in the United States, follow the vine and export the captured package to WirteShark to see:

The red part is the source IP, and the other party's operation to request the database (port 3306).

It is also possible that my server is just a DDoS attack node. Therefore, in order to maintain network security, the Trojan horse files must be processed in a timely manner.

Distributed Denial of Service attack (English means Distributed Denial of Service, DDoS for short) refers to multiple attackers in different locations simultaneously launching attacks on one or several targets, or one attacker controls multiple machines located in different locations And use these machines to attack the victims at the same time. Since the attack points are distributed in different places, this type of attack is called a distributed denial of service attack, in which there can be multiple attackers.
Schematic diagram of DDos attack

5. Prevention

The main reason for the server being attacked is that there are too many public network ports exposed, especially Redis6789, MySQL3306, and the server password is too simple.


1) Server

By modifying the /etc/ssh/sshd_configfile

  • Turn off password login, only allow secret key pair login
  • Change the default port of ssh to prevent the brute force library from being cracked
  • Disable root account direct login, open specific IP access

Such as:

AllowUsers    aliyun [email protected],[email protected]* 

# 拒绝 zhangsan、aliyun 帐户通过 SSH 登录系统
DenyUsers    zhangsan aliyun    

AuthorizedKeysFile .ssh/authorized_keys 
PubkeyAuthentication yes
RSAAuthentication yes

PasswordAuthentication no      

2) Application

  • Redis only allow local access, modify the default port, not exposed to all IP, Redis default bind a reason
  • MySQL only opens access to the required IP
  • Set firewall access rules for ports

If you want to be more secure, you can use springboard machines and bastion machines to access.

3) Backup

Back up data regularly, and back up snapshots regularly.

The above is the investigation process and some thoughts of a server being hacked.

If you want to learn about network detection and security audit of a website, you can learn about nmap: https://nmap.org/man/zh/index.html