Raspberry PI IoT Network Monitor
Work in Process ... see status & log
The goal of the SentinelPi is to watch-over our world of IoT smart devices and to learn about traffic patterns. An attempt to keep an eye on who is talking to who and to monitor behavior! Recipe will likely evolve as Network monitor tools are installed/tested and tuned.
Better ideas are always welcome!
SentinelPi: A Pi to watch over home network of smart devices
Original: December 2021
Last Update: 17 May 2023
*** 2023 RPi2, DietPi rebuild aftermath ... See log ***
- Network Traffic Monitor + Maltrail initially
As always, Pi recipes are basically notes-to-self, used to retrace steps for upgrades, rebuilds and such - possibly useful to others.
Initial tools - Maltrail & Network Traffic Monitor - seems to work well together!
Luckily, our smart 'Things' are isolated on a separate subnet behind a somewhat smart switch that provides SentinelPi a peek into ALL IoT traffic via a mirror port.
More to come but handy and works quite well!
Level of Difficulty & Prep Time
Still a bit advanced but not as complicated to install as it used to be ....
Probably a couple of hours worth of work plus some time to get things up, rolling and fine-tuned for your network,
Networking know-how (TCP/IP subnetting) and Linux knowledge will be required.
The SentinelPi environment is a combination of software, RPi hardware *and* cooperating network parts/features (built using old stuff laying around, of course!) ...
A semi-smart switch that supports port mirroring:
Like a TP-Link TL-SG108E*
V1 used a RPI3B+
V2 and beyond using a RPI2 (v1.1) - seems fine with workload
32g MicroSD (for starters)
Switched to 16g for V2
Step By Step
DietPi OS Install/Config
Initially running on RPi 3 w/ 32g SD that was sitting here doing nothing.
Switched to RPi2 w/ 16g SD ... seems to run fine (watches IOT traffic only) - will watch carefully.
Originally used RaspOS Lite per guide @ https://www.raspberrypi.com/documentation/computers/os.html.
Latest version uses DietPi as a base, much easier and better fit for our network - easier to maintain.
Initial Boot w/ Keyboard & Monitor attached but also easy via ssh
DietPi bootup/setup dialog
Display -> GPU Memory Split to 16 (server)
Language/Locale->Season to taste (timezone)
Security -> Set Hostname
Network Adapters -> Wifi Off, Ethernet set to Static (recommended)
DietPi Software (Season to taste here too, I'll prob use docker the next time around)
16 Build-Essential: GNU C/C++ compiler, development libraries and headers
17 Git: Clone and manage Git repositories locally
84 Lighttpd: Extremely lightweight webserver
103 DietPi-RAMlog: Makes /var/log a RAM disk, preserves file structure on reboot
104 Dropbear: Lightweight SSH server
130 Python 3: Runtime system, pip package installer and development headers
200 DietPi-Dashboard: Official lightweight DietPi web interface.
Couple of touchup installs (fav editor and some packages DietPi missed)
sudo apt-get install joe net-tools dnsutils netbase
<reboot-a-roo!> and switch to SSH for the rest of this
Finally turn off wireless stuff ... There's MORE than enough RF in our little network closet already!!
// reminder: rfkill survives through a boot ... to undo this use "unblock" //
sudo rfkill block wifi
sudo rfkill block bluetooth
Sing-a-long with https://github.com/stamparm/maltrail/blob/master/README.md#quick-start. The Whole README is very good!
Couple of leftover installs for this guy (some may already be there)
sudo apt-get install python-is-python3 libpcap-dev schedtool python3-pcapy
sudo pip3 install pcapy-ng
git clone --depth 1 https://github.com/stamparm/maltrail.git
Initial Testing ...
Install and fire up maltrail server (gui)
sudo python3 server.py &
Test via browser @ http://<ipaddress>:8338
Initial login is admin/changeme!
A couple of queries to get something logged (seem to be dangerous places! Test was in README)
ping -c 1 126.96.36.199
Refresh maltrail web interface to see logged threats
Complete Install and Auto-Start once we know this thing works!
Minimal config file touch-ups - /home/pi/maltrail/mailtrail.conf
Bounce the server to make sure you did that right! Log back into GUI after bounce
pkill -f server.py
python server.py &
Wile editing config, add a line to move Maltrail log files out of /var/log to avoid issues associated with DietPi Ramlog option (cleared maltrail logs)
Modify maltrail.config per following:
Add a little script to /etc/cron.monthly directory to delete logs older than 1 year (season to your own taste).
// Mine is called maltrail_log_cleanup.sh, I also "chmod -x'd it ... not sure if that was necessary )
REMEMBER! This is currently running in a terminal session! Not as a service, DON'T EXIT SSH OR CLOSE TERMINAL SESSION!
// See "Pull it all Together" section below - We will turn this into a Service that starts when the system boots //
Network Traffic Monitor Install/Config
Sing-a-long with https://www.technicallywizardry.com/raspberry-pi-network-monitor/ (read carefully)
Command line syntax & more in https://github.com/zaneclaes/network-traffic-metrics#readme
Installs (some of this may have been already installed with Maltrail, but we'll keep components independent)
// Also double check/set Eth promiscuous mode if you have been booting since Maltrail install //
I built a script to fire up eth0 monitor. Season tcpdump filters to taste (192.168.x.0/24) and append with a "&" to run in background ...
sudo python3 ./network-traffic-metrics.py -i eth0 -p 8001 "(src net 192.168.x.0/24 and not dst net 192.168.x.0/24) or (dst net 192.168.x.0/24 and not src net 192.168.x.0/24)"
Test via browser @ given ports. You should see statistics
Install Prometheus with help of https://prometheus.io/docs/prometheus/latest/getting_started/
cd /home/dietpi (don't install this under maltrail or network-monitor! I confused myself)
Download w/ help from https://pimylifeup.com/raspberry-pi-prometheus/
Check latest version (arm7 in my case) @ https://prometheus.io/download/
Insert version number and download via
$ wget https://github.com/prometheus/prometheus/releases/download/v2.37.5/prometheus-2.37.5.linux-armv7.tar.gz
Modify ./network-traffic-metrics/prometheus/prometheus.yml ... add localhost targets. Now looks like this ...
scrape_interval: 15s # How frequently to report
- job_name: 'network-traffic-metrics'
# Tped was here. target changed to localhost and target of 8001
- targets: ['localhost:8001'] # The Network Traffic Metrics IP/port
Test it out
Then surf to metrics via:
This step is now optional. We have another RPI on the home network running Grafana so I decided to use that for the Network Traffic Monitor dashboard - takes a little more load off of the RPI2.
If needed, SentinelPi can easily run Grafana ... it is included in DietPi optimized software - setup is easy:
Official docs @ https://grafana.com/docs/grafana/latest/installation/debian/.
Even better, a tutorial @ https://grafana.com/tutorials/install-grafana-on-raspberry-pi/
Surf to http://<ip>:3001 - login as admin/admin, it will force PW change
Configure: Add datasource and Network Traffic Monitor guys's dashboard.
Click on gear in Grafana UI:
Pull it all together and start everything @ boot
I'll want this contraption to come up by itself when the system boots. Here is the current order of events ...
Couple of crontab @reboot items (Append to file) ...
Put eth0 into promiscuous mode via crontab @reboot ...
# TPed was here, put interfaces into promisc mode
@reboot sudo ifconfig eth0 promisc
Network Traffic Monitor Auto Start: Stick this in crontab @ reboot also ... per recommendation in doc. Make sure python script is fully qualified!
# TPed was here again! Fire up network traffic monitor @ boot
@reboot sudo python3 /home/pi/network-traffic-metrics/network-traffic-metrics.py -i eth0 -p 8001 "(src net 192.168.xx.0/24 and not dst net 192.168.xx.0/24) or (dst net 192.168.xx.0/24 and not src net 192.168.xx.0/24)" &
Maltrail: Set to run as services so it auto starts after boot
sensor.py: This will run as a service. "maltrail-sensor.service" provided w/ install and I can take a hint!
First copy the provided maltrail -sensor.service file to where all the other .service files live!
sudo cp /home/pi/maltrail/maltrail-sensor.service /usr/lib/systemd/system/maltrail-sensor.service
Next, edit the .service file to set working directory to where I installed this thing. Now looks like this:
Description=Malicious traffic detection sensor https://github.com/stamparm/maltrail
server.py: Same bit for server, it will run as a service too - "maltrail-server.service" provided w/ install.
Copy the provided maltrail-server.service file to systemd services area...
sudo cp /home/pi/maltrail/maltrail-server.service /usr/lib/systemd/system/maltrail-server.service
Edit the .service file to set working directory for server.py:
Description=Malicious traffic detection server https://github.com/stamparm/maltrail
Grafana: If installed via dietpi-software, it will already be a servcie and managed
Prometheus Auto Start via https://sbcode.net/prometheus/prometheus-service/
This guy will be fired up from prometheus path so remember to have the correct prometheus.yml in the directory (/home/pi/prometheus/ in my case). Original yml was created under network-monitor thingy
Create service file @ /user/lib/systemd/system/prometheous.service
Description=Prometheus System Monitoring and Alerting software. More @ https://prometheus.io
Commit hand-built services to Dietpi memory so everything fires up @ boot and is controlled in one place
chmod service files ... I instinctively use 644 for these
sudo chmod 644 /usr/lib/systemd/system/maltrail-server.service
sudo chmod 644 /usr/lib/systemd/system/maltrail-sensor.service
sudo chmod 644 /usr/lib/systemd/system/prometheus.service
run dietpi-services utility to add Maltrail server and sensor and Prometeous:
// Use "Add Missing Service" option, pretty simple //
Re-Boot-A-Roo to check things out ....
Traffic Monitor data @ http://<ip>:8001
Prometheous should be @ http://<ip>:9090/metrics
Grafana (if installed) will be @ http://<ip>:3001
15 May 2023 - Recipe cleanup. Set/Check promiscuous mode at each step - in case of extra reboots during install/config
08 February 2023 - Prometheus is still dying: compaction failed.
I reluctantly appended a nightly reboot to cron: 0 0 * * * /sbin/reboot
25 January 2023 - Prometheus has been going into the ditch, rolled back to previous version (v2.31.5)
Sentinel Pi has been struggling with stats ... I'm wondering if Prometheus v2.37.5 has some arm/memory issues ...
log shows: prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
// I also saw some sort of memory "panic" in the journal but lost that //
... anyhoo ... I went back to the version of Prometheus I've used for past few years - prometheus-2.31.1.linux-armv7
07 January 2023 ... Whoops! Missing maltrail history!
It looks like maltrail and dietpi ramlog option may not work well together. The DietPi Ramlog option appears to clears ALL log files under /var/log on a regular basis - including maltrail's which also happen to be used for historical purposes - if you clink back in time using maltrail calendar, it uses /var/log/maltrail info.
Fix: Moved log files under the maltrail directory, they should be small (unless you are constantly under attack!). Will watch storage and rotate if necessary
Modify maltrail.config per following:
Added a little script to /etc/cron.monthly (maltrail_log_cleanup.sh)
03 Jan 2023 ... this thing has been running GREAT! Time to tweak it and mess it up!
Complete rebuild using dietpi and 16g sd .... and folded into recipe above. Install is MUCH easier under DietPi
Maltrail, Network Traffic monitor and prometheous are still installed by hand (we'll watch for docker for next round)
Grafana is still in the recipe but not used on our SentinelPi, we have another Grafana server on the home net
Now looking for web-based packet capture project, it would be nice to take a close look at traffic periodically
November 2022: Little tweak/update
Swapped in unused RPI2. It seems to be handling everything fine. Needed RPI3B+ for another project
Added script to watch over RPI2 throttling and such - https://github.com/tped/PiPower
updated/upgraded stuff (fingers crossed): sudo apt update && sudo apt upgrade
// Seemed to take care of OS & Grafana, the rest was left alone //
Added RPIMonitor to monitor the monitor! https://xavierberger.github.io/RPi-Monitor-docs/10_index.html
April 2022: Not bothering w/hostname resolution in Network Traffic Monitor anymore, I'm used to the IP addresses now ... It's like watching digital rain!
Prometheus has been going into the ditch periodically - running our of memory? or something?
looks like this...
I see some chatter @ https://github.com/prometheus/prometheus/issues/7378
Looks like there is also a prometheus update - 2.35 - via latest version (arm7) @ https://prometheus.io/download/. I scanned release notes from 2.32 to 2.35 for memory leaks and such ... maaaaybeeee?! I'll watch and update prometheous next month
ToDo: put together a SentinelPi Update procedure or script
March 2022: Little setback. The AP (from junk drawer in my workroom) seems to have given up the ghost - like a VERY HARD RESET whilst it was running!!! Unrelated to SentinelPi software & such ... BUT VERY ODD!! NetGear WRN2000 factory reset appears to have changed its Serial Number & Default password - they are now different from those printed on the router @ factory! No $h1t?!?!! I can get it working, but it's running @ about 1/2 speed w/ no apparent symptoms (WiFi or Ethernet errors, retransmits, et
February 2022: A few more ToDo's
Polished up this recipe, the whole contraption now starts when SentinelPi is booted
Things came up MUCH better this time around - I booted most of the closet when I switched over to AP for WiFi things
// I'm guessing there were some ARPs cached or something that caused some angst the first time //
ToDo: If/When I rebuild, it may be best to use DietPi for this one - better control over hardware and runtime world
Netmonitor: Still trying to figure out how to display hostnames rather than local IPs ... playing with -fqdn option - Not going to bother! I'm used to IP's now
January 2022. Project looks do-able ... a few hurdles and ToDo's and Fixed Stuff
Hostnames (as opposed to ip addresses) would be nice ... in both maltrail and traffic monitor.
Something is wrong (static route?) when running things via the AP ... e.g. HomePi SSH is broke! Maybe more! OK Now
Could I run even more monitors & tools? Wireshark-like thing? IDS, like snort? What else?
Make this thing restart-able. It's a B1tch to restart if I accidentally close my ssh session! Updating recipe as I go ...
Prometheus & Gafana were simple - just run them as systemd services
Maltrail May still need some work ... there was a systemd service template for the sensor but ... server needs to fireup at some point, I presume AFTER the sensor (which may be slow?). Firing up server.py in rc.local seems to be OK
Network-traffic-monitor agent/sensor thingies - created a couple of scripts for now, will need to be set up as services or cron @boot
Network configuration issues to deal with .... Promiscuous mode only works with THINGS that are wired ...
All Wired THINGS seems fine. SentinelPi sits on a port that receives mirrored Ethernet traffic from the port that heads to the ISP Router (Semi-Smart Switch)
WiFi THINGS: running second copy of agent /w different port to add in wifi traffic SEEMS to work ... BUT ... Promiscuous Mode only picks up broadcast traffic on our THINGS SSID ... Couple of things to try here
Need better understanding of WiFi Monitor Mode - what can a Pi do? Do I need external adapters, etc
No Guts/No Glory - add USB WiFi Dongle (I have a drawer-full of EDIMAX ). It looks like it's good to go for Monitor Mode BUT I still don't know if that would work
SentinelPi MAY need to be the main AP for these IoT world ... I think this would be easy (our IoT is on isolated SSID), but may take a bit more RPI horsepower
May be able to do RPi Monitor Mode tests on SuitiePi (RaspAP)
Investigate other probe-like configurations ... e.g. Netbook
Separate/Commercial WiFi AP is probably the best answer here. Proximity is a problem in our home, IoT devices seems to be spreading throughout the house. We already have areas with weak WiFi signals ... probably time for Mesh anyhoo
December 2021. Completed initial search for tools for this project.
Settled on maltrail + a couple of copies of Network Traffic Monitor python scripts running on a PI 3 I had laying around. PI has eth0 *and* wlan0 in promiscuous mode and is connected to our 'things' subnet via wifi and ethernet. The ethernet port is switch mirror of traffic destined for our ISP router and should pick up all traffic from wired 'things' on our network.
Initial software install/setup was all in foreground - no services, cron jobs or anything. I just left it all running in an ssh terminal session to see if it would all work together ... it seems to be! I've been watching cpu, disk space + whatever else I find to monitor. It seems to be working, I see traffic in the tools! I'm actually surprised!