Nuzzle is a lightweight, fast packet sniffer that looks for unexpected network scans and potential attacks.
It is intended for use by people who run online servers. (Regular home users probably won't have a use for it.)
Use Nuzzle to:
Monitor: Learn about the scans and attacks your server typically receives.
Detect: As part of an intrusion detection system (IDS), Nuzzle rapidly provides insight into the types of scans and attacks your system receives.
Prevent: With Fail2Ban, implement a simple but extremely effective intrusion prevention system (IPS).
Background
There are three common types of network attackers that you are likely to encounter online:
General Purpose Attackers. These people are attacking everyone and looking for the low hanging fruit. (You're online. They're online. They will attack you. It's not personal.) These attackers constantly scan and rescan the internet for potentially vulnerable syustems. Some only do one kind of scan, while others seek out any online system and queue it up for a large number of scans.
Blind Attackers. Performing reconnaissance and checking for vulnerabilities takes time. Sometimes it's faster to skip the reconnaissance. Blind attackers do not check if the victim's system could be vulnerable; they attack first and see if it worked. Similar to the general purpose attackers, these people attack everyone.
Directed Attackers. They don't want "any system"; they want your system. These people are often determined and focused. They may use customized attack methods to target your specific system. However, developing a novel attack takes time. Using Nuzzle as an IPS can slow down or discourage directed attackers.
With each of these attacker types, they often scan (or attack) a wide range of ports on your server. They don't just focus on the specific services that you allow through your firewall.
Nuzzle helps you identify the reconnaissance and attack packets. You can configure it with a list of "known open" services that you provide, so you don't log any permitted accesses. Any packets that attempt to access any other ports are logged by Nuzzle.
Enjoy!
Videos
Want to learn more? See the four-part "No-NOC Networking" video series on YouTube!
You need the source code: or or visit github. Save this file as "nuzzle.c". This file is open source, using a modified BSD license. (Free for non-commercial use.) You are welcome to look over the code. Any feedback is appreciated.
Compile it using g++. It's been tested with g++ and gcc versions 4.8 and 9.4, so any modern version should work without a problem.
g++ -Wall -o nuzzle nuzzle.c
This source code should compile very quickly (less than a second) and without any error messages. There are no dependencies on specialized libraries or third-party software. This simplies any code audits and software bill of materials (SBOM) dependency checks.
As an aside: Even though the code is C, the g++ compiler works better than the gcc compiler. Specifically, gcc doesn't known about boolean data types (stdbool was added later), while g++ natively supports booleans. g++ also provides many more error checks than gcc. This code should compile cleanly and without any warnings or errors.
Install the Nuzzle executable into /usr/local/bin:
sudo install ./nuzzle /usr/local/bin
Pre-compiled Executables?
I currently do not provide any pre-compiled executables since this becomes a maintenance nightmare. In effect, I'd need to support every library version on every platform. Even supplying a static executable requires cross-platform testing. Moreover, the installation and startup scripts vary by operating system and version. I'd need to provide repository versions for every release of Ubuntu, Debian, Fedora, and every other OS. This becomes a significant maintenance overhead. (A maintenance overhead that I'm not being paid to perform.)
It's much easier to just maintain the source code and let people download and compile it.
Besides, automatically locking down a system without customizing it for your specific server offerings will only lead to problems.
Windows?
What about Nuzzle for Windows? Nope, I'm not supporting that.
These days (2023), Windows accounts for over 70% of desktop operating systems. However, over 75% of servers run Linux or some version of Unix. Even Microsoft only uses their own operating system on about 6.6% of their servers. In my opinion, between the lack of server support and being designed as a single-user graphical environment, Windows is not a good choice for an internet-facing server.
In addition:
Windows doesn't use the same network functions as Linux, BSD, MacOS, and other Unix-like operating systems. Nuzzle's source code would need to be recreated for WinSock or rebuilt to support third-party dependencies like WinPcap.
Windows doesn't use syslog, so the logging function would need to be rewritten.
For automated attack mitigation, Nuzzle works with iptables and Fail2Ban. Neither iptables nor Fail2Ban exists for Windows.
Basically, "Nuzzle for Windows" would need to be a significant rewrite and development effort for relatively few servers.
Code Audits
The "nuzzle.c" program is provided as a single source code file.
There are no additional header files or third-party dependencies.
You are welcome to review the code.
The entire program only uses functionality that is part of the core gcc/g++ compiler distribution.
Because there are no other external dependencies, any audit based on dependencies can be completed almost instantly:
Software Bill of Materials (SBOM):
nuzzle.c
gcc or g++ (version 4.8 or later)
and nothing else.
There are no additional dependencies.
The code compiles cleanly and there are no memory leaks as detected by Valgrind and Electric Fence.
With regards to performance: Nuzzle has been tested at up to 10Gbps. There was no detected latency or dropped packets. (It should work on faster network connections, but we have not had the hardware for testing that configuration.)
Bugs?
If you encounter any bugs or things that should be fixed, please report it at Nuzzle's github.
Monitor
Usage
Exclusions
Simple Monitor
Nuzzle monitors for a wide range of packets that indicate scans or attacks. These include:
Unexpected TCP SYN packets.
Inbound UDP packets that are not in response to any outgoing UDP request.
Unexpected transport protocol requests, including UDPlite, SCTP, GRE, and IP encapsulation.
Any traceroute packet.
Any incoming ICMP packet, including pings and host redirections.
Use Nuzzle to establish a baseline for typicaly network activity and to watch for unexpected network traffic.
Usage
Run Nuzzle without any parameters to see the usage.
$ nuzzle
Nuzzle needs to be told which network interface to use. Use '-i list' to see the available interfaces. For example, my system lists:
$ nuzzle -i list
Available interfaces: eth0 eth1
In this case, 'eth0' is my external ethernet interface and 'eth1' is my backup interface. Your interface names will likely be different (and you may only have one listed), but select the external ethernet interface.
Running Nuzzle
Since the program needs to monitor the network interface, it must run as root. Use 'sudo' to tell it to run as root. The '-i eth0' identifies the network interface to monitor and '-t' says to print the time with each detected packet. It will output one line per packet. (If you use grep, sed, or awk to parse the output, then also use '-l' to enable line buffering.)
$ sudo nuzzle -t -i eth0
At any time you can use 'Control-C' to stop the program.
Since we didn't provide any permitted ports, every incoming TCP SYN, UDP, and ICMP packet will be logged. Here are a few sample log entries:
The time the packet was received. The '-t' parameter uses microseconds and shows the time in GMT.
The remote (client) network address, port, and protocol. For protocols that do not include ports (e.g., ICMP), it lists the type of packet ("8/icmp" is an echo request packet).
A separator "->" that indicates the packet direction from client to server. It also is a useful clue for scripts that parse the output.
The local (server) network address, with port and protocol as appropriate. (In this example, my server uses 10.207.45.17.) For protocols that do not use server-side ports, no port is listed.
The packet's time to live (TTL) field. For IPv6, the TTL value is the "hop limit" field.
A separator ":" that comes before any analysis description. It also is a useful clue for scripts that parse the output.
Keywords that describe the type of scan. These include the protocol name (icmp, tcp, udp, udplite, sctp, etc.) and any informative detections:
scanner: Any incoming packet that isn't explicitly permitted is listed as a scanner.
ping: An ICMP or ICMPv6 echo-request packet.
private: If the remote network address uses a private (non-routable) network range, then it is listed as "private".
traceroute: Any incoming packet with a TTL of 1 is identified as a traceroute packet.
redirect host: Congratulations! Someone tried to send you an ICMP redirect packet (often used for man-in-the-middle attacks). It logs the attack and where the attacker tried to redirect your traffic.
The keywords permit you to easily search through the logs for specific types of packets.
Excluding Expected Traffic
If you're running this on a server, then you should explicitly list the services you provide. For example, if you run a web server (80/tcp and 443/tcp) and a DNS server (53/udp), then you would use:
$ sudo nuzzle -t -i eth0 -T 80,443 -U 53
This tells Nuzzle to ignore (not log) any incoming packets to 80/tcp, 443/tcp, and 53/udp. (53/udp is only needed for someone hosting a DNS server. It's not needed for clients accessing DNS.)
Similarly, you can exclude the logging of incoming packets from known hosts. Such as:
This tells Nuzzle to exclude packets from 10.1.15.32, the entire subnet 192.168.0.0/16 (192.168.0.0 to 192.168.255.255), and the IPv6 range 2805:1bad:deed::/48 (any IPv6 address that begins with 2805:1bad:deed:).
Unexpected connection-oriented traffic (TCP) is easy to detect: any incoming TCP SYN request to a port that isn't in the permit list is logged. However, connectionless traffic (UDP) is a little more complicated. It is common for a server to send out a UDP packet and wait for a reply. This can happen during a DNS lookup, setting the computer's time, or streaming a video. For connectionless traffic, Nuzzle listens for any outbound UDP packets and temporarily marks the UDP port as permitted so that replies are not logged.
While Nuzzle won't log typical connections to these excluded ports, it will still list any identify traceroute packets. For example, if a scanner uses 'sudo traceroute -T -p 80', then they will be running traceroute against 80/tcp. Nuzzle will notice the traceroute and log the reconnaissance scan. Nuzzle will also record any instances of parasitic trace scans (paratrace).
About IDS
Configure
Monitor
Usage: Intrusion Detection System
An intrusion detection system (IDS) monitors the incoming network traffic for anything unexpected.
You can run Nuzzle on an existing server to measure the amount of scans and attacks the system receives.
Nuzzle also makes a great addition to any honeypot logs.
For my own systems, I run it on both the production servers and honeypots! This allows me to determine if the scans seen on the production server are similar to the noise received at the honeypot.
I compare the scans detected at the production server to the honeypot. I track the relative proportion of attacks since the production server blocks hostile scans while the honeypot does not. The production server should always receive fewer scans and attacks compared to the honeypot.
If the production server ever sees an increase in scans (compared to the typical rate seen when compared to the honeypot), then I probably have a directed and determined attacker.
If the honeypot ever sees an increase in scans (compared to the typical rate seen when compared to the production server), then I there is probably a new attack system out there. I can then determe the threat level that the attack method poses to the production server.
Configuration
Configuring the system as an IDS requires running Nuzzle as a daemon and the collecting logs.
1. Redirect Logging
When Nuzzle runs as a daemon, it logs to syslog. Create a configuration file that tells syslog to place the logs in /var/log/nuzzle.log.
sudo tee /etc/rsyslog.d/20-nuzzle.conf << EOF
template(name="nuzzlelog_list" type="list") {
property(name="timereported" dateFormat="year")
constant(value="-")
property(name="timereported" dateFormat="month")
constant(value="-")
property(name="timereported" dateFormat="day")
constant(value=" ")
property(name="timereported" dateFormat="hour")
constant(value=":")
property(name="timereported" dateFormat="minute")
constant(value=":")
property(name="timereported" dateFormat="second")
constant(value=" ")
property(name="hostname")
constant(value=" ")
property(name="app-name")
constant(value=":")
property(name="msg" spifno1stsp="on" ) # add space if \$msg doesn't start with one
property(name="msg" droplastlf="on" ) # remove trailing \n from \$msg if there is one
constant(value="\n")
}
if \$programname == 'nuzzle' then /var/log/nuzzle.log;nuzzlelog_list
& stop
EOF
Then restart the log handler: sudo service rsyslog restart
Typical syslog entries only record the day and time (not the year). This logs the full date and time of the log entry.
2. Enable Log Rotation
The /var/log/nuzzle.log file can get large. Rotate it as needed. This sample code rotates it daily and retains 7 days of logs.
You don't need to restart anything; logrotate will notice the file and start using it.
3. Run Nuzzle as a Daemon!
Add all of your permitted port numbers for TCP using -T and UDP using -U. For example, to permit web, email, and SSH, you would use:
sudo nuzzle -d -i eth0 -T 80,443,22
The "-d" says to run as a daemon and log to syslog. We don't include "-t" for the time since syslog already records the time.
If everything works, then you should be able to watch the logs using: tail -f /var/log/nuzzle.log
To make this permanent, add your Nuzzle start command to /etc/rc.local or wherever your system stores startup scripts. (The location varies based on your operating system distribution.)
There are over 4 million possible TCP sequence numbers (in hex: 00000000 to ffffffff). However, some appear more often than others. Repeated sequence numbers often indicate a botnet or scanning service.
The TCP sequence number is usually initialized with a random 32-bit value. You should see lots of sequence numbers with few sightings. If you have fewer than 4 million records, then some values will not be seen and many will only be seen once or twice.
Some scanning bots, including the Mirai botnet family, encode the destination's IPv4 address in the sequence number. For example, if your server's IP address is 10.198.99.90, then the sequence number will be "seq=0ac6635a". (That's the IPv4 address in hex.) This encoding reduces the tracking requirement by the scanner since it does not need to track which systems it scanned. Instead, the scanner looks at the sequence number in any reply packet to determine which system replied. This case is logged as "seq=ipv4.dst" and will likely be the most common sequence number logged by Nuzzle.
Some bots initialize the sequence number with constant values, such as zero ("00000000") or 100 ("00000064"). These are often the 2nd or 3rd most common sequence numbers observed by Nuzzle.
A few scanning bots encode the destination's IPv4 address in the sequence number but in reverse endian ordering. These is logged as "seq=4vpi.dst" (that's "ipv4" backwards).
About IPS
Configure
Monitor
Risks
Usage: Intrusion Prevention System
Network attackers often scan the system before attacking. (On my own server, I found that 70% of web attacks were preceeded by port scans.) If you can detect the initial scanning, then you can block the offending network address before it finds anything on the server to attack. On my systems, this reduced the overall volume of daily attacks by literally 99%. (A 70% reduction happened within hours. It increased to 90% after a week, and became 99% after 9 months.)
A Nuzzle-based intrusion prevention system (IPS) looks for any unexpected incoming packets and temporarily blocks the scanner from accessing anything on the server. When the scanner triggers a nuzzle log entry, it is denied access to everything on the server. This causes all subsequent scans to find nothing. If the attacker finds nothing, then they have nothing to attack. This will dramatically reduce the number of scans and attacks that your server receives, even on your permitted service ports.
Warning: Don't lock yourself out!
Automated blocking can be dangerous if you configure it incorrectly. Specifically, if you remotely log into the server, then an incorrect configuration may lock you out if you forget to include your login port. For example, if you login over ssh on 22/tcp, then be absolutely sure to permit 22/tcp (-T 22). Similarly, if you forget and 'ping' your server, then you will lock yourself out of the server for 15 minutes.
There are some options to prevent locking yourself out:
Alternate interface: For my own servers, I have two network interfaces (wan and lan). I only run Nuzzle on wan (-i wan). This means that I can always login over the lan.
Exclude logs: If you always login to your server from a specific network address, then you can add that address to the Fail2Ban filter. Edit /etc/fail2ban/filter.d/nuzzle.conf and add another entry to the ignoreregex section that identifies your network address.
Ignore bans: Fail2Ban supports a list of IP addresses and subnets that should never be banned. These can be added to the beginning of /etc/fail2ban/jail.local (create the file if it doesn't exist):
[DEFAULT]
ignoreip = 127.0.0.1 192.168.0.0/16
You can list individual network addresses or network ranges in CIDR format (with the "/mask").
Don't forget to restart Fail2Ban: sudo service fail2ban restart
Verify that they are marked as ignored:
Get the list of jails: sudo fail2ban-client status
Your output will vary based on your jails. Mine shows:
Status
|- Number of jail: 4
`- Jail list: nuzzle, slowlearner, sshd, ssl
Select any one of the listed jails and display the ignored IP settings. For example, I have a jail named 'slowlearner': sudo fail2ban-client get slowlearner ignoreip
You should see the list of ignored IP addresses, such as:
These IP addresses/networks are ignored:
|- 127.0.0.1
|- 192.168.0.0/16
Permit hosts: If you always connect to your service from the same network address, then you can use Nuzzle's '-H' parameter to list additional addresses that are permitted and not logged. (You can use -H multiple times, once for each permitted address.)
Permit local: Many cloud services forward external traffic to your internal virtual server. The clients will have external network addresses, but the server is on a private address such as "10.0.100.3". Nuzzle's -P parameter permits any clients that use private network addresses. This way, you can always access the server from inside the cloud.
Console access: Whether it is a physical server with a keyboard and monitor or a virtual server, there is always some kind of console for accessing the system. Nuzzle never blocks console access. If you lock yourself out, then you can always use the console to connect back in.
Configurating Nuzzle as an IPS
While a honeypot wants to receive everything, a prevention system wants to block all scans and attacks.
A typical IPS configuration includes:
Not responding to discovery scans.
Installing, patching, and configuring Fail2Ban in order to automatically block scanners and attackers.
1. Install Nuzzle as a Daemon
Follow the IDS instructions to configure Nuzzle for logging to syslog.
2. Install Fail2Ban
Fail2Ban monitors log files for patterns and watches for how often the pattern appears. When the occurance rate crosses a defined threshold, the associated IP address is temporarily blocked. (Step #5 will configure the pattern and define the rate.)
Note: Your installation method will vary based on your operating system. For example:
On Ubuntu: sudo apt install fail2ban
On Fedora: sudo yum install fail2ban
This will install Fail2Ban and iptables.
3. Patch Fail2Ban
Fail2Ban is designed to ban offending IP address. (Good.)
This normally stops the IP from appearing in the logs since the packets cannot reach the application.
However, Nuzzle is a packet sniffer. It sees packets even if the sender's IP address is banned.
When Fail2Ban tries to ban something twice, it will generate warning messages in /var/log/fail2ban.log. The warnings look like:
WARNING [nuzzle] <ip> already banned
These warnings just add noise to the logs.
Fail2Ban has different log levels (INFO, WARNING, etc.) According to Fail2Ban's changelog (see ver. 0.8.8, 2012/12/06), 'already banned' should be INFO, not WARNING. However, this is not the case in the current code. We need to fix it.
Edit /etc/fail2ban/fail2ban.conf.
You should see "loglevel = INFO" (that's a lot of noise).
Change it to "loglevel = NOTICE".
Edit /usr/lib/python3/dist-packages/fail2ban/server/actions.py
Search for "already banned".
Somewhere around line 500, you will see:
(Delete the middle 'else' and change the last 'else' from WARNING to INFO.)
Now the code matches the changelog's description. It doesn't show noise.
4. Stop discovery scans
How do attackers find things to attack? They perform a discovery scan!
In effect, they choose a network range to scan and send a packet to every IP address in that range.
Then they listen for any responses:
No reply? Assume nothing is at that network address and ignore it. You won't receive additional scans.
Found a service? There's a server there! They will add it to a queue for more detailed scanning and blind attacks.
Received a packet that indicates no service on that port? There may not be a service on that port, but there is a server at that IP address! They will it to a queue for more detailed scanning and blind attacks.
By not responding to typical discovery scans, you will dramatically reduce the number of scans and attacks received by your server.
Fail2Ban requires iptables, so it should already be installed.
Tell iptables and the network stack to not announce this server to attackers who are running discovery scans.
It's also good to put in some configurations for attack mitigation.
As root, use:
# For IPv4: Stop common discovery scans
sysctl -w net.ipv4.icmp_echo_ignore_all=1
iptables -I OUTPUT -p icmp --icmp-type destination-unreachable -j DROP
iptables -I OUTPUT -p tcp --tcp-flags ALL RST,ACK -j DROP
# For IPv6: Stop common discovery scans
sysctl -w net.ipv6.icmp.echo_ignore_all=1
ip6tables -I OUTPUT -p tcp --tcp-flags ALL RST,ACK -j DROP
ip6tables -I OUTPUT -p ipv6-icmp --icmpv6-type destination-unreachable -j DROP
ip6tables -I OUTPUT -p ipv6-icmp --icmpv6-type address-unreachable -j DROP
ip6tables -I OUTPUT -p ipv6-icmp --icmpv6-type port-unreachable -j DROP
# Blocking ICMPv6 echo-request breaks Teredo support, but almost nobody uses Teredo today.
# For man-in-the-middle prevention (these should only be enabled if you have a need for them to be enabled)
sysctl -w net.ipv4.conf.all.accept_redirects=0
sysctl -w net.ipv6.conf.all.accept_redirects=0
#### For DDoS prevention (these say ipv4.tcp but also apply to ipv6.tcp)
# SYN cookies avoid flooding the half-open TCP connection queue.
sysctl -w net.ipv4.tcp_syncookies=1
# Orphan connections consume 64K of unswappable memory.
# The default limit is very high; a DDoS can consume all RAM.
# Prevent DDoS by lowering it. If the limit is hit, all orphans are dropped and reset.
# Orphans are uncommon normally, but a high-speed attack can cause lots of them.
sysctl -w net.ipv4.tcp_max_orphans=256
# Reduce FIN_WAIT_2 state to 10 seconds max to prevent running out of sockets. (Default is usually 1 minute.)
sysctl -w net.ipv4.tcp_fin_timeout=10
# Maximum number of queued connections. It's typically 128, but a DDoS can overwhelm it.
# Servers should have a larger connection queue.
sysctl -w net.core.somaxconn=2048
To make this permanent:
There are a couple of "sysctl -w" commands. Add everything after the "sysctl -w" to the end of /etc/sysctl.conf. Then run: sudo sysctl -p
sudo tee -a /etc/sysctl.conf << EOF
# Stop common discovery scans
net.ipv4.icmp_echo_ignore_all=1
net.ipv6.icmp.echo_ignore_all=1
# For man-in-the-middle prevention (these should only be enabled if you have a need for them to be enabled)
net.ipv4.conf.all.accept_redirects=0
net.ipv6.conf.all.accept_redirects=0
# For DDoS prevention (these say ipv4.tcp but also apply to ipv6.tcp)
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_max_orphans=256
net.ipv4.tcp_fin_timeout=10
# Larger queue for servers
net.core.somaxconn=2048
EOF
sudo sysctl -p
Add the 'iptables' and 'ip6tables' lines to /etc/rc.local or wherever your operating system places startup scripts. (Or you can fool around with iptables-save and ip6tables-save. See your operating system's documentation on how to do this since the specific commands and file locations vary by Linux distibution.)
How do you know that these rules are working? You can temporarily watch all ICMP traffic for pings:
sudo tcpdump -l -n icmp or icmp6
You should see incoming ping requests (echo request) but no outgoing responses (echo reply).
5. Create the Fail2Ban Rules
Fail2Ban requires two configuration files per rule: a filter for scanning the log and a jail for implementing a ban.
First create the Fail2Ban filter. This filter parses the nuzzle.log file and identifies the IP address location on each log line. It also ignores a couple of log line messages.
sudo tee /etc/fail2ban/filter.d/nuzzle.conf << EOF
# Fail2Ban filter for Nuzzle
[INCLUDES]
# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf
[Definition]
_daemon = nuzzle
failregex = ^.*nuzzle: <HOST>[ \[].*\$
ignoreregex = ^.*nuzzle: message repeated.*\$
^.*permitted.*\$
^.*private.*\$
EOF
Then define the Fail2Ban jail. This jail watches for any unauthorized port scans or reconnaissance packets. If it sees any, it blocks access to this server (blocks all ports on all protocols) for 15 minutes.
sudo tee /etc/fail2ban/jail.d/nuzzle.conf << EOF
[nuzzle]
backend = pyinotify
enabled = true
# Block ALL ports on ALL protocols (tcp, udp, etc.)
# And DROP (don't send destination unrechable).
banaction= iptables-allports[blocktype=DROP,protocol=all]
port = 0:65535
protocol = all
filter = nuzzle
logpath = /var/log/nuzzle.log
# If seen even 1 time, ban for 15 minutes (60*15=900 seconds)
maxretry = 1
findtime = 10
bantime = 900
EOF
Now you can restart Fail2Ban:
sudo service fail2ban restart
The system should automatically ban any scanners that touch your system. Each ban lasts 15 minutes. That is, 15 minutes after the last time the IP address tried to scan the system. (Some attackers will repeatedly try to scan your system even after they have been banned.)
6. Test!
Make sure it works. It should not take much time for the scans and attacks to show up.
To monitor the bans in real-time, use:
This shows both log files in real-time. You should see the nuzzle.log entries followed by a Fail2Ban entries.
After 15 minutes, the logs should show the bans being automatically removed by Fail2Ban.
Monitor Fail2Ban
Fail2Ban includes some great features for monitoring the effectiveness of these rules.
For example:
See all active jails
Get a list of all of the current jails:
sudo fail2ban-client status
You should see a "Jail list" line that includes "nuzzle". For example:
Status
|- Number of jail: 2
`- Jail list: nuzzle, sshd
See banned addresses
Get a list of all of the currently banned network addresses for the jail named 'nuzzle':
sudo fail2ban-client get nuzzle banip
See banned durations
Get a list of all of the currently banned network addresses for the jail named 'nuzzle' with timestamps:
It shows the offending IP address, the time each was first banned, and the time when each ban will expire. Most of your entries should say "+ 900" (seconds), indicating a 15-minute ban. Values greater than 900 means that the IP address repeatedly tried to scan the server even though it was already banned. Values larger than 900 denote persistent attacking IP addresses.
You can sort the list by ban duration, which permits identifying the most persistent addresses:
sudo fail2ban-client get nuzzle banip --with-time | sort +4 -n
In this example (from my logs), 89.248.165.66 has been banned nearly 8 hours. (28,320 seconds is 7 hours 52 minutes.) I then searched through my logs for this IP address (grep 89.248.165.66 /var/log/nuzzle.log) and identified 249 scan attempts, with a new scan every 1-5 minutes. This specific network address is associated with a self-proclaimed "researcher" called Recyber. Recyber regularly scans large chunks of the internet and attacks any services that it discovers.
$ host 89.248.165.66
66.165.248.89.in-addr.arpa domain name pointer recyber.net.
Recyber's web site claims that they assist researchers and universities, but they don't identify who runs Recyber, where they are located, how to contact them, or which universities they work with. A preliminary search for universities and academic publications that say they work with Recyber turned up nothing. (Recyber acquired the domain name in 2021. From 2003-2006, the domain was related to an unrelated finance and real estate service, which does appear in academic papers.)
Interestingly, the scan pattern used by Recyber seems to be coordinated with similar patterns from China, Russia, and North Korea. You'll see this if you watch your logs. After a few hours or days, Recyber will stop and Chang Way (AS57523; associated with Russian, China, and North Korea) will start up doing the same type of slow-but-steady scans for hours or days. Then they will change addresses again to some anonymous cloud provider or back to Recyber. Based on this correlation, it appears that Recyber is a front for attackers in Russia, China, and/or North Korea.
IPS Risks
With automated network address blocking, it is very easy for an attacker to generate packets with fake source addresses.
This would cause Nuzzle to log the packets and Fail2Ban to block the forged addresses.
A malicious attacker could craft packets that cause the server to block access to required services.
For example, if your system uses Google's public DNS servers for name resolution (8.8.8.8), then an attacker could generate a fake scan that appears to come from host "8.8.8.8". This would cause Nuzzle to log the attack as if it came from 8.8.8.8 and Fail2Ban would block the network address. As a result, your server would be unable to access DNS.
To mitigate this attack vector, Nuzzle permits you to specify multiple network addresses and ranges that should always be permitted. For example:
This sample usage will montitor eth0 (-i eth0), log to stdout using timestamps (-t), and permit any packets from host 8.8.8.8, the IPv4 network range 12.34.56.0/24, and the IPv6 network range 2805:1bad:deed::/48. While the attacker can still generate the forged packets, they will not result in a ban. Moreover, an attacker cannot use these forged packets to scan the server since the attacker will never see the reply packet.
In general, you should only use -H to permit network addresses that you trust and are essential.
In addition to using Nuzzle's "-H" parameter, you can configure Fail2Ban so that it never blocks the permitted network addresses. (It's a good idea to use both Nuzzle's -H parameter and configure fail2ban to never block your local and trusted addresses.)
Edit "/etc/fail2ban/fail2ban.conf"
The configuration file should have a section at the top called "[DEFAULT]". Add an 'ignoreip' parameter that lists the permitted network ranges. For example:
Whether you run Nuzzle on a honeypot or a production server, there are some interesting patterns that you might notice. For example:
The longer you are online, the more scans and attacks you will receive. A brand new honeypot on a previously-unused IP address (no inherited traffic from a previous server) might initially see 300-500 scans and attacks per hours. However, after a day or two, it will increase to 1,000 per hour. After a month, it might be 5,000 per hour.
Established, popular online servers (production services) can receive 10,000 scans and attacks per minute if they don't do any filtering to deter scans and attacks. The general formula for determining the attack volume is:
(Duration + Popularity) × Services - Filtering = Volume of scans and attacks
Duration: The longer a server is online, the more likely it will be discovered by scanners and attackers. This usually peaks after being online for six months.
Popularity: The easier it is to find the service, the more it will be attacked. Very popular servers are scanned and attacked more often than smaller, unknown services. Unlike the duration, there is no apparent upper limit to the volume impact from popularity; more popular means more attacks.
Services: The more services you provide, the more it will attract scanners and attackers. They will not just target your existing services; they will scan the entire server and attack anything they find.
Filters: The more defenses you deploy the fewer scans and attacks you will receive.
Look in the logs for the repeat offenders and subnets that keep performing scans. These identify known-hostile providers. By looking at their hostnames and subnet providers, you can quickly identify who is behind the scans and attacks. These include:
Hostile providers: Every packet from them is malicious. These are often subnets owned and operated by nation-states or organized crime.
Mostly-hostile providers: These include cloud and ISP services where over 90% of their packets are scans and attacks. You might occasionally see a regular (non-hostile) user, but most of their bandwidth is hostile.
Bulletproof hosting providers: These are typically cloud and co-location providers that cater to attackers. They are called bulletproof because, regardless of how much you complain, these providers do little or nothing to stop the abuse that eminates from their networks. Hostile organizations can operate from these networks with impunity.
Various "research" organizations: These are companies that constantly attack everyone on the internet. They often claim to be performing 'research' or are being used by researchers. These companies include Shodan, Censys, BinaryEdge, Rapid7, Recyber, and many others. These researchers will tell you that they are not "attacking" because they don't have the intent to compromise your system. However, there is no 'intent' flag in the packet. The packets sent by these researchers look just like the packets from the known-hostile services. The only difference between the researchers and the known-hostiles is that the researchers attack more often. The differences between these hostile "researchers" and legitimate researchers include:
Real research uses ethical methods. Constantly attacking other online services and performing unauthorized audits is unethical. When applying tests to other people, good researchers get permission first. Shodan and the other network scanners do not have informed consent.
There's a difference between illegal and illicit. While their actions may not be forbidden by law (illegal), they certainly are unethical and go against social norms and values (illicit).
Many of these "researchers" don't publish their work. In the case of Shodan, they do make their scan findings public, but they don't tell the victims of any potential problems. People who use Shodan to find vulnerabilities will learn about your problems before you learn about them.
They often do not follow the scientific process. Collecting data without a testable hypothesis is problematic. I've seen nothing to indicate that they use external validation. Moreover, most of these hostile scanners lack transparency in their processes.
Related to transparency, many of these groups don't identify themselves when doing the scans. Often, they hide behind cloud providers in order to anonymize their attacks. I think this is because they know that what they are doing is hostile, undesirable, and unethical. They don't want people to know that they are behind the attacks.
Their opt-out methods are either unavailable or do not work. In the case of BinaryEdge, they actively criticize anyone who doesn't want to be attacked by their service.
For newer systems and servers that are not very popular, the amount of ICMP traffic will be near the amount of TCP scans. However, older services, popular services, and systems running multiple services will likely see much more TCP than ICMP.
If you don't block ICMP echo request (ping) packets, they you will see a flood of pings coming from Amazon's AWS cloud. They will come in every minute and be (mostly) synchronized to the second. This isn't "Amazon" scanning you; this is some Amazon cloud customer who has a large number of IP addresses. Oh, and it's not just AWS; it's AWS in Asia.
If you do block ICMP echo request (ping) packets, then you won't see the Amazon ping floods. Instead, you'll notice a wide range of network addresses sending "10 pings at a time", all in one second. The packet contents (ICMP payload) includes a timestamp and your server's IP address. This is likely someone trying to do time-based geolocation. (Data travels at nearly the speed of light. By measuring the round-trip latency, they can estimate the maximum distance. By measuring from a wide range of network locations, they can triangulate your server's location to within a few kilometers.)
IPv4 vs IPv6
The vast majority of scans will come from IPv4. In general, IPv6 accounts for a small fraction of the scans and attacks that you will receive. If you use IPv6, then you'll only be scanned if the attackers can find you. How do they find you? They usually look in DNS for an IPv6 address (AAAA record) associated with your hostname.
If you don't have a DNS record with an IPv6 entry, then you probably won't be scanned over IPv6.
If you don't have an easy way to find your domain name, then they won't find your DNS record and you probably won't be scanned over IPv6.
Even if your server supports IPV6 and you have an IPv6 entry, very few attackers and scan software packages support IPv6. Even if they can scan you, it won't be anywhere near the same volume as IPv4.
Since most attackers use hostname-based lookups, how do they find your hostname? Maybe you're running a well-known or popular web site. Maybe someone posted a link with your domain name in it on some public forum. While unlikely, maybe they downloaded a list of registered domain names from somewhere. More likely is that you have a TLS certificate that lists your hostname. (E.g., https://{your IPv4 address}/ returns a TLS certificate or they connect to your mail server's submission port using TLS. In either case, the TLS certificate lists your domain name.) They will download the TLS certificate over IPv4, query the DNS record for the hostname(s) listed in the TLS certificate, and then scan any resulting AAAA records.
Even if they find your IPv6 address, it may take weeks or months before someone sets up a regular ICMPv6 ping scanner. Based on my honeypot tests (repeatedly tested in early 2023):
It will take less than a day for someone to query your IPv4 system for a TLS certificate (HTTPS or mail submission), query your DNS record for an IPv6 (AAAA) entry, and begin scanning your IPv6 address.
When they find your IPv6 address, the scans and attacks will account for about 0.5% of your hostile traffic. This is before the ping scanners start up.
The ICMPv6 ping scanners may take weeks or months to appear (2 weeks to 2 months), but when they appear the ICMPv6 ping scans will account for half of your daily scans and attacks. This doesn't mean that the volume of IPv4 attacks decreases; it means there is an additional high volume from IPv6.
While IPv6 pings may account for over 300,000 pings per week, TCP scans will slowly increase from 0.5% to 4% of the total TCP scans. (I've seen this take over 6 months.)
Disabling IPv4 pings will quickly reduce the number of pings that your server receives over IPv4.
However, disabling IPv6 pings will have zero impact on the flood of IPv6 ping requests. Even though your server isn't answering, the flood will continue unabated.