Attacked by an Amazon EC2 hosted botnet (Part 2) DNS server under attack.

So earlier this evening things started to slow down again and upon investigation discovered the attackers were now hitting my DNS servers.  I applied the same firewall rules to it as I did to the main server.

Amazon in the meantime responded and told me they had dealt with one of the IP’s for my original report.  Another email responding to a question informed me that for additional IPs I have to submit a fresh report.

I don’t fancy having to do this manually each time so decided to give them all the data and do it automatically.  I was tempted to do a separate report for each IP but that would have been churlish of me.

So, how to automate this.

The first thing to note is that because I am blocking the entire IP range of Amazon EC2 I can no longer use application logs such as Nginx, Named or Apache etc.  I will have to use the firewall to log this.

Here is how I do that.

The first thing to do is create a new chain that logs and drops.  It does nothing else and has no actual entries in it.  This chain will be called from other chains when a match is found.

iptables -N LOG_DROP
iptables -A LOG_DROP -j LOG --log-level warning --log-prefix "INPUT-DROP:"
iptables -A LOG_DROP -j DROP

The first line creates the new chain.  The second line does the logging.  The third line does the drop.  It then returns to the calling chain.

UPDATE:  I was dropping all connections from Amazon EC2.  This turned out to not be a fantastic idea because some services were using EC2 for mail delivery.  I have changed the log and drop chain to this now.

iptables -A INPUT -p tcp -m multiport --destination-port 80,443 -j DROP

Even though they are hitting the DNS server now, it is UDP traffic and they do not have enough hosts (approx 8,000) to make a dint on my servers performance.

So then I have to flush the amazon-ec2 chain

iptables -F amazon-ec2

Then re-add all the ip ranges using this new format.

iptables -I amazon-ec2 -s <IP> -j LOG_DROP -w

When a match is found, rather than the previous format command which just drops the connection, it now calls the LOG_DROP chain.

What I get in my /var/log/messages log file is a line for each dropped connection like this.

Jan 25 23:51:25 **** kernel: INPUT-DROP:IN=eth0 OUT= MAC=*:*:*:*:* SRC=34.223.47.190 DST=*.*.*.* LEN=84 TOS=0x00 PREC=0x00 TTL=232 ID=29013 PROTO=UDP SPT=28184 DPT=53 LEN=64

To issue a report I copy all the log entries to a new file and rotate the log file (to make clean matches in future parsing easier).

I take a representative sample of say 5 seconds from the file, so Amazon can see the frequency of the attack and a sample of the ports etc.

I then extract ALL the IP addresses that are taking part in the attack by using this command;

cat /var/log/messages | grep 'INPUT-DROP' | sed -rn -e 's,.* SRC=([0-9.]+).*,\1,p' | sort -t . -k1,1n -k2,2n -k 3,3n -k 4,4n | uniq

So after this I can send the first report.

Now I need to monitor the log file for additional IP addresses that are attacking but remove any that I have already reported.  Then create a new report using a new section from the /var/log/message file and only the new IP addresses.  I will provide an update after this happens.