Suppose you have a nginx
container running by Docker
on your server and you noticed some DDoS traffics.
The first thing you might think about is to block the attacker’s IP addresses. You might use iptables
to block the identified IP addresses. However, this might not work as you expected because docker isn’t very friendly with iptables.
Another aspect is that DDoS means the attacker can use lots of different IP addresses to attack. In fact, there are even tools that help attackers perform DDoS via Tor network easily. So blocking IPs won’t help much.
A popular solution is to add some sort of rate limiting on requests. Nginx
provides some features to help people configure that (link1, link2).
1. nginx
To set it up, first you need to create a nginx.conf
file on your host containing content like this:
http {
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
server {
location /login {
limit_req zone=one;
}
}
}
This means each unique IP address can only login once per second.
However, one caveat is that you might find some legit requests got rate limited. This can happen if there are OPTIONS
requests coming to your server before the actual POST
requests. So in some cases, you want to only rate limit on GET
or POST
requests.
This can be easily done by changing the above configs slightly as (ref):
http {
map $request_method $limit {
default "";
POST $binary_remote_addr;
}
limit_req_zone $limit zone=one:10m rate=1r/s;
server {
location /login {
limit_req zone=one;
}
}
}
2. docker
Since you are running nginx
inside a docker container, you will need to use docker volume to mount the local nginx.conf
file to the config inside the container.
This can be done easily with (ref):
$ docker run --name my-custom-nginx-container -v /host/path/nginx.conf:/etc/nginx/nginx.conf:ro -d nginx
3. monitoring
Once you set up everything and restart your container, you can curl
test to see if the rate limiting is actually working. You should be able to find something similar as this in your nginx
logs by running docker logs <name for the nginx container>
:
2022/07/07 00:07:16 xxx.xxx.xxx.xxx "POST /login HTTP/1.1" 200 ...
2022/07/07 00:07:16 xxx.xxx.xxx.xxx "POST /login HTTP/1.1" 503 ...
2022/07/07 00:07:17 xxx.xxx.xxx.xxx [error] 40#40: *1181 limiting requests, excess: 0.689 by zone "one", request: "POST /login HTTP/1.1", ...
xxx.xxx.xxx.xxx
will be your own IP address used for testing.
Another thing you will want to know is whether you are blocking legit requests, to do this you will need to analyze the logs. An example script is like this:
#!/usr/bin/python
import sys, re
if len(sys.argv) != 3:
raise ValueError('Number of arguments should be 3! Got {} instead'.format(len(sys.argv)))
# parse the command line arguments
stdout_log = sys.argv[1]
stderr_log = sys.argv[2]
# get the rate limited error log lines
stderr_lines_rate_limited = [ line for line in open(stderr_log, 'r') \
if 'limiting requests, excess' in line and 'by zone "one"' in line]
# identify the rate limited IP addresses from the error logs
ip_pattern = re.compile(r'(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})')
rate_limited_ips = []
for line in stderr_lines_rate_limited:
ip = ip_pattern.search(line)[0]
if ip not in rate_limited_ips:
rate_limited_ips.append(ip)
print("Found {} rate-limiited IPs.".format(len(rate_limited_ips)))
# locate the nginx stdout log lines with the rate-limited IPs
stdout_lines_with_rate_limited_ip = [ line for line in open(stdout_log, 'r') \
if any(ip in line for ip in rate_limited_ips)]
print("Found {} requests from rate-limited ips.".format(len(stdout_lines_with_rate_limited_ip)))
# find the requests that are rate limited
rate_limited_requests = []
for line in stdout_lines_with_rate_limited_ip:
pattern = re.compile(r'"POST /login HTTP/\d.\d" 503')
if pattern.search(line):
rate_limited_requests.append(line)
print("Found {} rate-limiteed requests.".format(len(rate_limited_requests)))
# identify the requests that are likely from a DDoS attack
ddos_count = 0
for line in rate_limited_requests:
if <conditions that can help you identify the attacker>:
ddos_count +=1
print("Found {} DDoS-like requests.".format(ddos_count))
To run the script, you will first need to get the stdout
and stderr
nginx logs to two files. You can do it via docker logs nginx 1> out.txt 2> err.txt
and then run the script with command python3 script.py out.txt err.txt
.
The output will be like this:
Found 138 rate-limited IPs.
Found 567 requests from rate-limited ips.
Found 209 rate-limited requests.
Found 209 DDoS-like requests.
If the 3rd number does not equal to the 4th number, you might want to dig into the logs further to see if there are legit requests being rate limited.