0x9f9f

Posted on Jul 08, 2022Read on Mirror.xyz

How to prevent DDoS attack on nginx?

Suppose you have a nginx container running by Docker on your server and you noticed some DDoS traffics.

The first thing you might think about is to block the attacker’s IP addresses. You might use iptables to block the identified IP addresses. However, this might not work as you expected because docker isn’t very friendly with iptables.

Another aspect is that DDoS means the attacker can use lots of different IP addresses to attack. In fact, there are even tools that help attackers perform DDoS via Tor network easily. So blocking IPs won’t help much.

A popular solution is to add some sort of rate limiting on requests. Nginx provides some features to help people configure that (link1, link2).

1. nginx

To set it up, first you need to create a nginx.conf file on your host containing content like this:

http {
  limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;

  server {
    location /login {
      limit_req zone=one;
    }
  }
}

This means each unique IP address can only login once per second.

However, one caveat is that you might find some legit requests got rate limited. This can happen if there are OPTIONS requests coming to your server before the actual POST requests. So in some cases, you want to only rate limit on GET or POST requests.

This can be easily done by changing the above configs slightly as (ref):

http {
  map $request_method $limit {
    default         "";
    POST            $binary_remote_addr;
  }
  limit_req_zone $limit zone=one:10m rate=1r/s;

  server {
    location /login {
      limit_req zone=one;
    }
  }
}

2. docker

Since you are running nginx inside a docker container, you will need to use docker volume to mount the local nginx.conf file to the config inside the container.

This can be done easily with (ref):

$ docker run --name my-custom-nginx-container -v /host/path/nginx.conf:/etc/nginx/nginx.conf:ro -d nginx

3. monitoring

Once you set up everything and restart your container, you can curl test to see if the rate limiting is actually working. You should be able to find something similar as this in your nginx logs by running docker logs <name for the nginx container>:

2022/07/07 00:07:16 xxx.xxx.xxx.xxx "POST /login HTTP/1.1" 200 ...
2022/07/07 00:07:16 xxx.xxx.xxx.xxx "POST /login HTTP/1.1" 503 ...
2022/07/07 00:07:17 xxx.xxx.xxx.xxx [error] 40#40: *1181 limiting requests, excess: 0.689 by zone "one", request: "POST /login HTTP/1.1", ...

xxx.xxx.xxx.xxx will be your own IP address used for testing.

Another thing you will want to know is whether you are blocking legit requests, to do this you will need to analyze the logs. An example script is like this:

#!/usr/bin/python

import sys, re

if len(sys.argv) != 3:
  raise ValueError('Number of arguments should be 3! Got {} instead'.format(len(sys.argv))) 

# parse the command line arguments
stdout_log = sys.argv[1]
stderr_log = sys.argv[2]

# get the rate limited error log lines
stderr_lines_rate_limited = [ line for line in open(stderr_log, 'r') \
  if 'limiting requests, excess' in line and 'by zone "one"' in line]

# identify the rate limited IP addresses from the error logs
ip_pattern = re.compile(r'(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})')
rate_limited_ips = []
for line in stderr_lines_rate_limited:
    ip = ip_pattern.search(line)[0]
    if ip not in rate_limited_ips:
      rate_limited_ips.append(ip)
print("Found {} rate-limiited IPs.".format(len(rate_limited_ips)))

# locate the nginx stdout log lines with the rate-limited IPs
stdout_lines_with_rate_limited_ip = [ line for line in open(stdout_log, 'r') \
  if any(ip in line for ip in rate_limited_ips)]
print("Found {} requests from rate-limited ips.".format(len(stdout_lines_with_rate_limited_ip)))

# find the requests that are rate limited
rate_limited_requests = []
for line in stdout_lines_with_rate_limited_ip:
  pattern = re.compile(r'"POST /login HTTP/\d.\d" 503')
  if pattern.search(line):
    rate_limited_requests.append(line)
print("Found {} rate-limiteed requests.".format(len(rate_limited_requests)))

# identify the requests that are likely from a DDoS attack
ddos_count = 0
for line in rate_limited_requests:
  if <conditions that can help you identify the attacker>:
    ddos_count +=1
print("Found {} DDoS-like requests.".format(ddos_count))

To run the script, you will first need to get the stdout and stderr nginx logs to two files. You can do it via docker logs nginx 1> out.txt 2> err.txt and then run the script with command python3 script.py out.txt err.txt.

The output will be like this:

Found 138 rate-limited IPs.
Found 567 requests from rate-limited ips.
Found 209 rate-limited requests.
Found 209 DDoS-like requests.

If the 3rd number does not equal to the 4th number, you might want to dig into the logs further to see if there are legit requests being rate limited.