Raspberry PI IoT Network Monitor

Initially running on RPi 3 w/ 32g SD that was sitting here doing nothing. 
Switched to RPi2 w/ 16g SD ... seems to run fine (watches IOT traffic only)  -  will monitor load carefully. 

Using DietPi as a base - easier to maintain.

1. Download and flash latest Dietpi image (Note image variations - arm v6/v7/v8).  Raspberry Pi Imager is now easiest.  

2. Initial Boot w/ Keyboard & Monitor attached but also easy via ssh  (login as root/dietpi initially)

3. DietPi bootup/setup dialog

   • Change passwords

   • Disable Serial/UART

4. DietPi-config

   • Language/Locale->Season to taste (timezone, kb, etc )

   • Security -> Set Hostname

   • Network Adapters -> Wifi Off, Ethernet set to Static (recommended)

5. DietPi Software (Season to taste here too, I'll prob use docker the next time around)

   • Build-Essential: GNU C/C++ compiler, development libraries and headers (V3: See below, manual install on latest Dietpi)

   • tcpdump:  command-line network traffic analyzer   

   • Git: Clone and manage Git repositories locally

   • Lighttpd: Extremely lightweight webserver 

   • DietPi-RAMlog: Makes /var/log a RAM disk, preserves file structure on reboot

   • OpenSSH: Feature-rich SSH server with SFTP and SCP support

   • Python 3: Runtime system, pip package installer and development headers

   • DietPi-Dashboard: Official lightweight DietPi web interface.

   • Samba Server: Feature-rich SMB/CIFS server

6. Couple of touchup installs (fav editor and some packages DietPi missed)
   sudo apt-get install joe net-tools dnsutils netbase build-essential rfkill

7. <reboot-a-roo!> and switch to SSH and user dietpi for rest of this

8. Finally turn off wireless stuff ... There's MORE than enough RF in our little network closet already!!
   // Prob done in dietpi-config.  Reminder:  rfkill survives through a boot ... to undo this use "unblock" //
   sudo rfkill block wifi
   sudo rfkill block bluetooth

Sing-a-long with https://www.technicallywizardry.com/raspberry-pi-network-monitor/ (read carefully)
Command line syntax & more in https://github.com/zaneclaes/network-traffic-metrics#readme

1. Installs (some of this may have been already installed with Maltrail, but we'll keep components independent)
   // Also double check/set eth0 promiscuous mode if you have been booting since Maltrail install // 

cd /home/dietpi
sudo pip3 install argparse prometheus_client

git clone https://github.com/zaneclaes/network-traffic-metrics.git

cd ./network-traffic-metrics

2. I built a script to fire up eth0 monitor.  Season tcpdump filters to taste (192.168.x.0/24) and append with a "&" to run in background ...
   sudo python3 ./network-traffic-metrics.py -i eth0 -p 8001 "(src net 192.168.x.0/24 and not dst net 192.168.x.0/24) or (dst net 192.168.x.0/24 and not src net 192.168.x.0/24)" &

3. Test via browser @ given ports.  You should see statistics
   http://<ip>:8001

This step is now optional.  We have another RPI on the home network running Grafana so I decided to use that for the Network Traffic Monitor dashboard - takes a little more load off of the RPI2.

If needed, SentinelPi can easily run Grafana ... it is included in DietPi optimized software - setup is easy:

1. To install Grafana via Dietpi (runs as a service)
   run dietpi-software and select/install:
   77  Grafana: platform for analytics and monitoring 

• More info @ https://dietpi.com/docs/software/hardware_projects/#grafana

• Official docs @ https://grafana.com/docs/grafana/latest/installation/debian/.  

• Even better, a tutorial @ https://grafana.com/tutorials/install-grafana-on-raspberry-pi/

1. Surf to http://<ip>:3001 - login as admin/admin,  it will force PW change

2. Configure:  Add datasource and Network Traffic Monitor guys's dashboard.  

   • Click on gear in Grafana UI:

     1. Add Datasource->Choose Prometheus and set server to http://<ip of server>:9090

     2. Add Dashboard:  Clink on "+" then import Network Traffic Dashboard by number (12619)
        Once installed, set dashboard datasource to Prometheus (from step 1)

October 2024 - V3 Rebuild and attempt to add some packet capture 

•  Power supply died!   8 Year old Canakit, constant use.  Not bad

• Decided to do annual rebuild on a spare RPI3B+ with bigger SD (32g)
  Will play with adding IDS or Packet Capture 

• Recipe updates along the way

  • Dietpi:  Updated from v9.2.1 to v9.7.1 (arm8)

  • Dietpi Config:  Added SMB Server.   May be moving more stuff in/out

  • Maltrail:   Updated from v .053 to v 074
    one feed error during initial run:
    [x] something went wrong during remote data retrieval ('https://www.talosintelligence.com/documents/ip-blacklist')

Fix (right or wrong):  I downloaded ip-blocklist from url above and added to dprk.txt in ~/maltrail/trails/custom

• • Prometheus:   Updated from v2.37.5 (arm7) to v2.54.41 (arm64)

Oct 2023 - Couple of notes

• ServerPi now runs Grafana in our world.  Takes a little load off of RPi2.   Install/Config steps remain in above recipe  

  • Also noticed that Datasource may now be found under "Connections" in latest version of grafana

15 May 2023 - Recipe cleanup.  Set/Check promiscuous mode at each step - in case of extra reboots during install/config

08 February 2023 - Prometheus is still dying:  compaction failed.  

I reluctantly appended a nightly reboot to cron:  0 0 * * * /sbin/reboot

25 January 2023 - Prometheus has been going into the ditch, rolled back to previous version (v2.31.5)

Sentinel Pi has been struggling with stats ... I'm wondering if Prometheus v2.37.5 has some arm/memory issues ...

log shows:  prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
// I also saw some sort of memory "panic" in the journal but lost that //

...  anyhoo ... I went back to the version of Prometheus I've used for past few years - prometheus-2.31.1.linux-armv7

07 January 2023 ... Whoops!  Missing maltrail history!

1. It looks like maltrail and dietpi ramlog option may not work well together.  The DietPi Ramlog option appears to clears ALL log files under /var/log  on a regular basis - including maltrail's which also happen to be used for historical purposes - if you clink back in time using maltrail calendar, it uses /var/log/maltrail info. v2.4.41
   Fix:  Moved log files under the maltrail directory, they should be small (unless you are constantly under attack!).  Will watch storage and rotate if necessary

   • Modify maltrail.config per following:

# Directory used for log storage 
# TPed was here.  Moved the logs to maltrail directory so we don't bump
# heads with DietPi-RAMlog options (clears /var/log) 
# LOG_DIR $SYSTEM_LOG_DIR/maltrail
LOG_DIR /home/dietpi/maltrail/log 

1. • Added a little script to /etc/cron.monthly (maltrail_log_cleanup.sh)

#!/bin/bash

# Delete Maltrail log files older than 1 year

find /home/dietpi/maltrail/log -mtime +365 -delete

03 Jan 2023 ... this thing has been running GREAT!  Time to tweak it and mess it up!

• Complete rebuild using dietpi and 16g sd .... and folded into recipe above.  Install is MUCH easier under DietPi

• Maltrail, Network Traffic monitor and prometheous are still installed by hand (we'll watch for docker for next round)

• Grafana is still in the recipe but not used on our SentinelPi, we have another Grafana server on the home net

• Now looking for web-based packet capture project, it would be nice to take a close look at traffic periodically

November 2022:  Little tweak/update

1. Swapped in unused RPI2.  It seems to be handling everything fine.   Needed RPI3B+ for another project 

2. Added script to watch over RPI2 throttling and such - https://github.com/tped/PiPower

3. updated/upgraded stuff  (fingers crossed):   sudo apt update && sudo apt upgrade
   // Seemed to take care of OS & Grafana, the rest was left alone //

4. Added RPIMonitor to monitor the monitor!  https://xavierberger.github.io/RPi-Monitor-docs/10_index.html

April 2022:  Not bothering w/hostname resolution in Network Traffic Monitor anymore, I'm used to the IP addresses now ... It's like watching digital rain!

Prometheus has been going into the ditch periodically - running our of memory?  or something?
looks like this...

SentinelPi prometheus[392]: ts=2022-04-20T03:43:07.200Z caller=head.go:803 level=info component=tsdb msg="Head GC completed" duration=19.600777ms

SentinelPi prometheus[392]: ts=2022-04-20T03:43:07.367Z caller=db.go:830 level=error component=tsdb msg="compaction failed" err="compact head: head memory truncate: truncate chunks.HeadReadWriter: mmap, size 134217728: cannot allocate memory"

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.264Z caller=compact.go:518 level=info component=tsdb msg="write block" mint=1650420006283 maxt=1650427200000 ulid=01G12NFH6X53RW9510QTTDGS9S duration=666.434323ms

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.295Z caller=head.go:803 level=info component=tsdb msg="Head GC completed" duration=22.080236ms

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.298Z caller=db.go:830 level=error component=

tsdb msg="compaction failed" err="compact head: head memory truncate: truncate chunks.HeadReadWriter: write dat

a/chunks_head/000545: file already closed"

....

yada yada

....

prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

SentinelPi systemd[1]: prometheus.service: Failed with result 'exit-code'.

Prometheus version:

./prometheus --version

prometheus, version 2.31.1 (branch: HEAD, revision: 411021ada9ab41095923b8d2df9365b632fd40c3)

 build user:       root@b013bc8edd0b

 build date:       20211105-20:27:14

 go version:       go1.17.3

 platform:         linux/arm

I see some chatter @ https://github.com/prometheus/prometheus/issues/7378

Updating everything

apt update
apt upgrade

Looks like there is also a prometheus update - 2.35 - via latest version (arm7) @ https://prometheus.io/download/.  I scanned release notes from 2.32 to 2.35 for memory leaks and such ... maaaaybeeee?!   I'll watch and update prometheous next month

ToDo:  put together a SentinelPi Update procedure or script

---------------------

March 2022:  Little setback.  The AP (from junk drawer in my workroom) seems to have given up the ghost - like a VERY HARD RESET whilst it was running!!!   Unrelated to SentinelPi software & such ... BUT VERY ODD!!  NetGear WRN2000 factory reset appears to have changed its Serial Number & Default password - they are now different from those printed on the router @ factory!  No $h1t?!?!!  I can get it working, but it's running @ about 1/2 speed w/ no apparent symptoms (WiFi or Ethernet errors, retransmits, et

February 2022:  A few more ToDo's

• Polished up this recipe, the whole contraption now starts when SentinelPi is booted

• Things came up MUCH better this time around - I booted most of the closet when I switched over to AP for WiFi things
  // I'm guessing there were some ARPs cached or something that caused some angst the first time //

• ToDo:  If/When I rebuild, it may be best to use DietPi for this one - better control over hardware and runtime world

• Netmonitor:  Still trying to figure out how to display hostnames rather than local IPs ... playing with -fqdn option - Not going to bother!  I'm used to IP's now

January 2022.   Project looks do-able ... a few hurdles and ToDo's and Fixed Stuff

1. Hostnames (as opposed to ip addresses) would be nice  ... in both maltrail and traffic monitor.  

2. Something is wrong (static route?) when running things via the AP ... e.g. HomePi SSH is broke!  Maybe more! OK Now

3. Could I run even more monitors & tools?  Wireshark-like thing?  IDS, like snort?  What else?

4. Make this thing restart-able.  It's a B1tch to restart if I accidentally close my ssh session!  Updating recipe as I go ... 

   • Prometheus & Gafana were simple - just run them as systemd services

   • Maltrail May still need some work ... there was a systemd service template for the sensor but ... server needs to fireup at some point, I presume AFTER the sensor (which may be slow?).  Firing up server.py in rc.local seems to be OK

   • Network-traffic-monitor agent/sensor thingies - created a couple of scripts for now, will need to be set up as services or cron @boot

5. Network configuration issues to deal with .... Promiscuous mode only works with THINGS that are wired ...

   • All Wired THINGS seems fine.  SentinelPi sits on a port that receives mirrored Ethernet traffic from the port that heads to the ISP Router (Semi-Smart Switch)

   • WiFi THINGS:  running second copy of agent /w different port to add in wifi traffic SEEMS to work ... BUT ... Promiscuous Mode only picks up broadcast traffic on our THINGS SSID ...  Couple of things to try here

     1. Need better understanding of WiFi Monitor Mode - what can a Pi do?  Do I need external adapters, etc

        • Good Article @ https://networkengineering.stackexchange.com/questions/3774/using-wireless-cards-in-promiscuous-mode/3787#3787

        • No Guts/No Glory - add USB WiFi Dongle (I have a drawer-full of EDIMAX ).  It looks like it's good to go for Monitor Mode BUT I still don't know if that would work

     2. SentinelPi MAY need to be the main AP for these IoT world ... I think this would be easy (our IoT is on isolated SSID), but may take a bit more RPI horsepower 

        • May be able to do RPi Monitor Mode tests on SuitiePi (RaspAP)

        • Investigate other probe-like configurations ... e.g. Netbook

     3. Separate/Commercial WiFi AP is probably the best answer here.  Proximity is a problem in our home, IoT devices seems to be spreading throughout the house. We already have areas with weak WiFi signals ... probably time for Mesh anyhoo

December 2021.  Completed initial search for tools for this project.   

Settled on maltrail + a couple of copies of Network Traffic Monitor python scripts running on a PI 3 I had laying around.   PI has eth0 *and* wlan0 in promiscuous mode and is connected to our 'things' subnet via wifi and ethernet.  The ethernet port is switch mirror of traffic destined for our ISP router and should pick up all traffic from wired 'things' on our network.  

Initial software install/setup was all in foreground - no services, cron jobs or anything.  I just left it all running in an ssh terminal session to see if it would all work together ... it seems to be!  I've been watching cpu, disk space + whatever else I find to monitor.  It seems to be working, I see traffic in the tools!   I'm actually surprised! 

--------- Odd's n ends from attempt to add packet capture service *** this is a mess *** ---

Actually got Suricata working on a RPI3B+ but it really wasn't enough Horsepower.  Will revisit when one of my RPI4s is available.  Notes below are just the tip of the iceberg ... lots to learn about Suricata and IDS

Quick review of packet capture/options for SentinelPi

1. Suricata IDS (RPI step by step @ https://jufajardini.wordpress.com/2021/02/15/suricata-on-your-raspberry-pi/)

   • Install Suricata and  updater
     sudo apt install suricata suricata-update 

   • Run updater
     sudo suricata-update

   • Edit /etc/suricata/suricata.yml

     1. Set HOME_NET to match IP address scheme

# TPed was here:  HOME_NET set to IOT subnet

address-groups:

   HOME_NET: "[192.168.xx.0/24]"

1. • 2. Adjust default-rule-path to match where updater  put rules

# TPed was here:  Update default rule path after running update
#default-rule-path: /etc/suricata/rules
default-rule-path: /var/lib/suricata/rulesrules

1. • 3. We need to move suricata's eve.yml file since our SentinalPi uses Diepi's Ramlog 

sudo mkdir -p /mnt/dietpi_userdata/suricata

joe /etc/suricata/suricata.yaml

# TPed was here:  Moved eve.json to dietpi_userdata so we can use dietpi ramlog

 - eve-log:

     enabled: yes

     filetype: regular #regular|syslog|unix_dgram|unix_stream|redis

     filename: /mnt/dietpi_userdata/suricata/eve.json

1. • Test it out!

sudo suricata -T -c /etc/suricata/suricata.yaml

Output should look a little like this:

17/10/2024 -- 07:37:34 - <Info> - Running suricata under test mode

17/10/2024 -- 07:37:35 - <Notice> - This is Suricata version 6.0.10 RELEASE running in SYSTEM mode

17/10/2024 -- 07:38:14 - <Notice> - Configuration provided was successfully loaded. Exiting.

2. gui:  evebox (Doc @ https://evebox.org/docs/install/ )

   • Download and install .deb (no  apt that I could see) from https://evebox.org/files/release/latest/ 

cd ~/
wget https://evebox.org/files/release/latest/evebox-0.18.2-arm64.deb
sudo dpkg -i evebox-0.18.2-arm64.deb
sudo apt-get install -f

2. •  Output will look a little like this:

Selecting previously unselected package evebox.

(Reading database ... 26960 files and directories currently installed.)

Preparing to unpack evebox-0.18.2-arm64.deb ...

Unpacking evebox (1:0.18.2) ...

Setting up evebox (1:0.18.2) ...

+ USERNAME=evebox

+ HOMEDIR=/var/lib/evebox

+ /usr/bin/getent passwd evebox

+ test -e /usr/sbin/adduser

+ /usr/sbin/adduser --system --home /var/lib/evebox --group --disabled-login evebox

Adding system user `evebox' (UID 102) ...

Adding new group `evebox' (GID 109) ...

Adding new user `evebox' (UID 102) with group `evebox' ...

Creating home directory `/var/lib/evebox' ...

2. • See if we missed anything:

sudo apt --fix-broken install

2. • Configure:  Use SQlite, 

     1. Double check dietpi-software and install SQlite, if necessary

[*] 87  SQLite: Persistent single-file database system

2. • 2.  Modify the EveBox configuration file (/etc/evebox/evebox.yaml) ... 

        • ...to use SQLite and limit retention:

# TPed was here:  Switch DB to SQlite

database:

 type: sqlite

 sqlite:

   filename: /mnt/dietpi_userdata/evebox/evebox.sqlite

retention:

 days: 7

 size: "15 GB"

2. • 2. • ... to allow access from any host (as long as you are editing)

http:

 ## By default, EveBox binds to localhost. Uncomment this line to open

 ## it up.

 # TPed was here:  Allow access from any host

 host: "0.0.0.0"

2. • 3. Give it a test

sudo evebox server --data-directory /mnt/dietpi_userdata/evebox --input /var/log/s

uricata/eve.json --host 0.0.0.0 --port 5636 --sqlite

2. • 3. • Watch for admin  user and password during  the test ... remember these!

2024-10-17 09:18:49  WARN evebox::server::main: Username/password authentication is required, but no user

s exist, creating a user

2024-10-17 09:18:50  WARN evebox::server::main: Created administrator username and password: username=adm

in, password=N0tTh1sEa5y!

2. • 3. • Check web GUI, you should see stuff in "Events" tab

https://<IP>:5636/

Pull it all together

sudo dietpi-services -> add both

geoip:  https://www.maxmind.com/en/geoip2-services-and-databases
https://evebox.org/docs/server/configuration 

________________________

2. • Configure, compile
     cd $HOME/suricata-7.0.7/
     ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --enable-nfqueue --enable-lua
     make
     
     

   • more to come
     sudo apt-get suricata

   •  222  sudo apt-get install suricata

   •  223  sudo joe /etc/suricata/suricata.yaml  

   •  224  sudo joe /etc/suricata/suricata.yaml
     (update home netsowrk)sudo apt-get uninstall eve 

   •  225  systemctl status

   •  226  sudo systemctl status

   •  227  sudo systemctl status | grep sur

   •  228  sudo apt-get install evebox

   •  229  wget https://evebox.org/files/release/latest/evebox-0.18.2-arm64.deb

   •  230  sudo dpkg -i evebox-0.18.2-arm64.deb  

   •  231  sudo apt-get install -f

   •  232  sudo reboot

   •  233  sudo systemctl status

   •  234  sudo systemctl status | grep sur

   •  235  sudo systemctl status suricata

   •  236  sudo systemctl status evebox

   •  237  dietpi-services  

   •  238  sudo dietpi-services  

   •  239  sudo joe /etc/evebox/evebox.yaml  

   •  240  cat /etc/evebox/evebox.yaml  

   •  241  cat /etc/evebox/evebox.yaml  

   •  242  sudo reboot

   •  243  sudo dietpi-services  

   •  244  history

   • 
     
   • 
     

   •   

   • 
     
   • 
     

3. Simple bash script that starts tcpdump and writes to a share - can be reviewed using wireshark on my desktop 
   See Below

#!/bin/bash

# Check if an IP address was provided

if [ -z "$1" ]; then

 echo "Usage: $0 <IP_ADDRESS>"

 exit 1

fi

# Set the capture directory and ensure it exists

CAPTURE_DIR="/mnt/dietpi_userdata/captures"

mkdir -p "$CAPTURE_DIR"

# Define the IP address and the base name for capture files

IP_ADDRESS="$1"

TIMESTAMP=$(date +"%Y%m%d_%H%M%S")

BASE_FILENAME="$CAPTURE_DIR/capture_${IP_ADDRESS}_${TIMESTAMP}.pcap"

# Run tcpdump with elevated permissions

sudo tcpdump -i any host "$IP_ADDRESS" -w "$BASE_FILENAME"

echo "Capture started for IP $IP_ADDRESS ... <ctl>C to stop"