Raspberry PI IoT Network Monitor

Work in Process ...  see status & log

The goal of the SentinelPi is to watch-over our world of IoT smart devices and to learn about traffic patterns.  An attempt to keep an eye on who is talking to who and to monitor behavior!    Recipe will likely evolve as Network monitor tools are installed/tested and tuned.

Fun one! 

Better ideas are always welcome!

SentinelPi:  A Pi to watch over home network of smart devices

Original:  December 2021
Last Update:  17 May 2023
*** 2023 RPi2, DietPi rebuild aftermath ...  See log ***
- Network Traffic Monitor + Maltrail initially

Project Status

As always,  Pi recipes are basically notes-to-self, used to retrace steps for upgrades, rebuilds and such -  possibly useful to others.   

Initial tools - Maltrail & Network Traffic Monitor - seems to work well together!  

Luckily, our smart 'Things' are isolated on a separate subnet behind a somewhat smart switch that provides SentinelPi a peek into ALL IoT traffic via a mirror port.   

More to come but handy and works quite well!

Level of Difficulty & Prep Time

Still a bit advanced but not as complicated to install as it used to be ....

Probably a couple of hours worth of work plus some time to get things up, rolling and fine-tuned for your network,   

Networking know-how (TCP/IP subnetting) and Linux knowledge will be required.   


The SentinelPi environment is a combination of software, RPi hardware *and* cooperating network parts/features  (built using old stuff laying around, of course!) ...
// *Note:  Affiliate Links, I may get kickback from Amazon if you click/buy! //

Network hardware looks a bit like this:

Note:  ALL IoT traffic MUST flow through the SentinelPi for these monitors.  This recipe relies on network design (isolated subnet) and  capabilities (Port Mirroring). There are alternative approaches - See  Internet_Traffic_Monitor_Approaches.

Step By Step

DietPi OS Install/Config

Initially running on RPi 3 w/ 32g SD that was sitting here doing nothing. 
Switched to RPi2 w/ 16g SD ... seems to run fine (watches IOT traffic only)  - will watch carefully.  

Originally used RaspOS Lite per guide @ https://www.raspberrypi.com/documentation/computers/os.html
Latest version uses DietPi as a base, much easier and better fit for our network - easier to maintain.

Maltrail Install/Config

Sing-a-long with https://github.com/stamparm/maltrail/blob/master/README.md#quick-start.  The Whole README is very good!

netstat -i
Kernel Interface table
eth0      1500   151201      0      0 0         70372      0      0      0 BMPRU
lo       65536       14      0      0 0            14      0      0      0 LRU

# Directory used for log storage 
# TPed was here.  Moved the logs to maltrail directory so we don't bump
# heads with DietPi-RAMlog options (clears /var/log) 
LOG_DIR /home/dietpi/maltrail/log 


# Delete Maltrail log files older than 1 year

find /home/dietpi/maltrail/log -mtime +365 -delete

// See "Pull it all Together" section below - We will turn this into a Service that starts when the system boots //

Network Traffic Monitor Install/Config

Sing-a-long with https://www.technicallywizardry.com/raspberry-pi-network-monitor/ (read carefully)
Command line syntax & more in https://github.com/zaneclaes/network-traffic-metrics#readme

cd /home/dietpi
sudo apt-get install tcpdump
sudo pip3 install argparse prometheus_client

git clone https://github.com/zaneclaes/network-traffic-metrics.git

cd ./network-traffic-metrics

Prometheus Install/Config

Install Prometheus with help of https://prometheus.io/docs/prometheus/latest/getting_started/

Then surf to metrics via:

Grafana Install/Config

This step is now optional.  We have another RPI on the home network running Grafana so I decided to use that for the Network Traffic Monitor dashboard - takes a little more load off of the RPI2.

If needed, SentinelPi can easily run Grafana ... it is included in DietPi optimized software - setup is easy:

Pull it all together and start everything @ boot

I'll want this contraption to come up by itself when the system boots.   Here is the current order of events ... 

Re-Boot-A-Roo to check things out ....

Traffic Monitor data @ http://<ip>:8001
Prometheous should be @ http://<ip>:9090/metrics
Grafana (if installed) will be @ http://<ip>:3001 



Oct 2023 - Couple of notes

15 May 2023 - Recipe cleanup.  Set/Check promiscuous mode at each step - in case of extra reboots during install/config

08 February 2023 - Prometheus is still dying:  compaction failed.  

I reluctantly appended a nightly reboot to cron:  0 0 * * * /sbin/reboot

25 January 2023 - Prometheus has been going into the ditch, rolled back to previous version (v2.31.5)

Sentinel Pi has been struggling with stats ... I'm wondering if Prometheus v2.37.5 has some arm/memory issues ...

log shows:  prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
// I also saw some sort of memory "panic" in the journal but lost that //

...  anyhoo ... I went back to the version of Prometheus I've used for past few years - prometheus-2.31.1.linux-armv7

07 January 2023 ... Whoops!  Missing maltrail history!

# Directory used for log storage 
# TPed was here.  Moved the logs to maltrail directory so we don't bump
# heads with DietPi-RAMlog options (clears /var/log) 
LOG_DIR /home/dietpi/maltrail/log 


# Delete Maltrail log files older than 1 year

find /home/dietpi/maltrail/log -mtime +365 -delete

03 Jan 2023 ... this thing has been running GREAT!  Time to tweak it and mess it up!

November 2022:  Little tweak/update

April 2022:  Not bothering w/hostname resolution in Network Traffic Monitor anymore, I'm used to the IP addresses now ... It's like watching digital rain!

Prometheus has been going into the ditch periodically - running our of memory?  or something?
looks like this...

SentinelPi prometheus[392]: ts=2022-04-20T03:43:07.200Z caller=head.go:803 level=info component=tsdb msg="Head GC completed" duration=19.600777ms

SentinelPi prometheus[392]: ts=2022-04-20T03:43:07.367Z caller=db.go:830 level=error component=tsdb msg="compaction failed" err="compact head: head memory truncate: truncate chunks.HeadReadWriter: mmap, size 134217728: cannot allocate memory"

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.264Z caller=compact.go:518 level=info component=tsdb msg="write block" mint=1650420006283 maxt=1650427200000 ulid=01G12NFH6X53RW9510QTTDGS9S duration=666.434323ms

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.295Z caller=head.go:803 level=info component=tsdb msg="Head GC completed" duration=22.080236ms

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.298Z caller=db.go:830 level=error component=

tsdb msg="compaction failed" err="compact head: head memory truncate: truncate chunks.HeadReadWriter: write dat

a/chunks_head/000545: file already closed"


yada yada


prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

SentinelPi systemd[1]: prometheus.service: Failed with result 'exit-code'.

Prometheus version:

./prometheus --version

prometheus, version 2.31.1 (branch: HEAD, revision: 411021ada9ab41095923b8d2df9365b632fd40c3)

 build user:       root@b013bc8edd0b

 build date:       20211105-20:27:14

 go version:       go1.17.3

 platform:         linux/arm

I see some chatter @ https://github.com/prometheus/prometheus/issues/7378

Updating everything

apt update
apt upgrade

Looks like there is also a prometheus update - 2.35 - via latest version (arm7) @ https://prometheus.io/download/.  I scanned release notes from 2.32 to 2.35 for memory leaks and such ... maaaaybeeee?!   I'll watch and update prometheous next month

ToDo:  put together a SentinelPi Update procedure or script


March 2022:  Little setback.  The AP (from junk drawer in my workroom) seems to have given up the ghost - like a VERY HARD RESET whilst it was running!!!   Unrelated to SentinelPi software & such ... BUT VERY ODD!!  NetGear WRN2000 factory reset appears to have changed its Serial Number & Default password - they are now different from those printed on the router @ factory!  No $h1t?!?!!  I can get it working, but it's running @ about 1/2 speed w/ no apparent symptoms (WiFi or Ethernet errors, retransmits, et

February 2022:  A few more ToDo's

January 2022.   Project looks do-able ... a few hurdles and ToDo's and Fixed Stuff

December 2021.  Completed initial search for tools for this project.   

Settled on maltrail + a couple of copies of Network Traffic Monitor python scripts running on a PI 3 I had laying around.   PI has eth0 *and* wlan0 in promiscuous mode and is connected to our 'things' subnet via wifi and ethernet.  The ethernet port is switch mirror of traffic destined for our ISP router and should pick up all traffic from wired 'things' on our network.  

Initial software install/setup was all in foreground - no services, cron jobs or anything.  I just left it all running in an ssh terminal session to see if it would all work together ... it seems to be!  I've been watching cpu, disk space + whatever else I find to monitor.  It seems to be working, I see traffic in the tools!   I'm actually surprised! 

Initial Fumblings

--------- Odd's n ends from initial work ... I may still salvage something from here ---

Tested a bunch of stuff.   Attempting to make a little gizmoPi that watches over our quickly growing collection of IoT "things".

Redoo-a-Roo #2! Latest will be above this from now on!


NTOPng - I forgot about this guy! It works OK, but pretty much becomes useless after a short period of time (10 mins) . It is nice, but not worth subscription price for our home needs.

Install - started @ https://packages.ntop.org/ and downloaded RPI via:

wget https://packages.ntop.org/RaspberryPI/apt-ntop_1.0.190416-469_all.deb

sudo dpkg -i apt-ntop_1.0.190416-469_all.deb

sudo apt-get install ntopng nprobe n2n

Moving on to installation guide @ https://www.ntop.org/guides/ntopng/what_is_ntopng.html#installing-on-linux

Install DOES permanently install ... ntopng will start when the system is rebooted an sits right on top of grafana (port 3000).

// I'll shut it down manually when working on maltrail/netmon stuff //


Maltrail - Sing-along-with https://computingforgeeks.com/setup-maltrail-malicious-traffic-detection-system-on-linux/

Maltrail Readme is VERY good too ... find it @ https://github.com/stamparm/maltrail/blob/master/README.md

Install Notes - rough for me? So I can retrace my steps

Test using mirrored IoT switch port & promisc wifi ... We'll see!?

// Seemed to run ok, but ... can't recall why I did a re-doo-a-roo but I did! //

----------- Part 2 - IoT traffic Monitor PLUS maltrail ---------

RaspOS lite as base, more network traffic tools then maltrail

Then surfing to metrics via:


netstat -i // Make sure interfaces are in promiscuous mode //

cd ./network-traffic-metrics

sudo python3 ./network-traffic-metrics.py "(src net 192.168.xx.0/24 and not dst net 192.168.xx.0/24) or

(dst net 192.168.xx.0/24 and not src net 192.168.xx.0/24)" -i wlan0 -p 8000 &

cd prometheus

./prometheus --config.file=prometheus.yml &

test python via http://<ip>:8000

test prometheus via http://<ip>:9090

Bookmark collection

More info @ https://haxf4rall.com/2018/06/13/maltrail-malicious-traffic-detection-system/ (some popup email collector, tho)

InternetPi on top ... from https://github.com/geerlingguy/internet-pi

target tools @ https://sectools.org/ + https://sectools.org/tag/traffic-monitors/ (old)

Net Monitor w/ Prometheus @ https://www.technicallywizardry.com/raspberry-pi-network-monitor/

Static Domain servers @ https://www.thegeekpub.com/18336/change-the-raspberry-pi-dns-settings/

Delete stuff from Prometheus in https://www.shellhacks.com/prometheus-delete-time-series-metrics/