Raspberry PI IoT Network Monitor

Initially running on RPi 3 w/ 32g SD that was sitting here doing nothing. 
Switched to RPi2 w/ 16g SD ... seems to run fine (watches IOT traffic only)  - will watch carefully.  

Originally used RaspOS Lite per guide @ https://www.raspberrypi.com/documentation/computers/os.html. 
Latest version uses DietPi as a base, much easier and better fit for our network - easier to maintain.

1. Download and flash latest Dietpi image (Note image variations - arm v6/v7/v8).  Raspberry Pi Imager is now easiest.  

2. Initial Boot w/ Keyboard & Monitor attached but also easy via ssh

3. DietPi bootup/setup dialog

   • Change passwords

   • Disable Serial/UART

4. DietPi config

   • Display -> GPU Memory Split to 16 (server)

   • Language/Locale->Season to taste (timezone)

   • Security -> Set Hostname

   • Network Adapters -> Wifi Off, Ethernet set to Static (recommended)

5. DietPi Software (Season to taste here too, I'll prob use docker the next time around)

   • 16   Build-Essential: GNU C/C++ compiler, development libraries and headers

   • 17   Git: Clone and manage Git repositories locally

   • 84   Lighttpd: Extremely lightweight webserver

   • 103  DietPi-RAMlog: Makes /var/log a RAM disk, preserves file structure on reboot

   • 104  Dropbear: Lightweight SSH server

   • 130  Python 3: Runtime system, pip package installer and development headers

   • 200  DietPi-Dashboard: Official lightweight DietPi web interface.

6. Couple of touchup installs (fav editor and some packages DietPi missed)
   sudo apt-get install joe net-tools dnsutils netbase 

7. <reboot-a-roo!> and switch to SSH for the rest of this

8. Finally turn off wireless stuff ... There's MORE than enough RF in our little network closet already!!
   // reminder:  rfkill survives through a boot ... to undo this use "unblock" //
   sudo rfkill block wifi
   sudo rfkill block bluetooth

Sing-a-long with https://github.com/stamparm/maltrail/blob/master/README.md#quick-start.  The Whole README is very good!

1. Couple of leftover installs for this guy (some may already be there)
   
   sudo apt-get install python-is-python3 libpcap-dev schedtool python3-pcapy
   sudo pip3 install pcapy-ng
   git clone --depth 1 https://github.com/stamparm/maltrail.git
   
   

2. Initial Testing ...

   • Check Promiscuous mode (you should see "P" in flags)

netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0      1500   151201      0      0 0         70372      0      0      0 BMPRU
lo       65536       14      0      0 0            14      0      0      0 LRU

2. • Fire up sensor ... it will take a bit as it updates trails, feeds or whatevers
     cd maltrail
     sudo python3 sensor.py &

   • Install and fire up maltrail server (gui)
     sudo python3 server.py &

   • Test via browser @ http://<ipaddress>:8338

     1. Initial login is admin/changeme!

     2. A couple of queries to get something logged (seem to be dangerous places!  Test was in README)
        ping -c 1 136.161.101.53  
        nslookup morphed.ru

     3. Refresh maltrail web interface to see logged threats

   • Complete Install and Auto-Start once we know this thing works!

     1. Minimal config file touch-ups - /home/pi/maltrail/mailtrail.conf 

        • Change password:

          1. Create a new password like this:
             echo -n '<NewStrongPassword>' | sha256sum | cut -d " " -f 1 

          2. Edit maltrail.conf and plant your new password (:0: at end) in the USERS section, replace the default password (changeme!)
             # User entries (username:sha256(password):UID:filter_netmask(s))
             # Note(s): sha256(password) can be generated on Linux with: echo -n 'password' | sha256sum | cut -d " " -1
             #          UID >= 1000 have only rights to display results (Note: this moment only functionality implemented at the c
             #          filter_netmask(s) is/are used to filter results
             USERS
               admin:1ee0cd0713ba2b4fde13881618e0995ee2f560c41fb7b2d195847e5a018fc03c2:0:                  # NewStrongPassword
             #   local:9ab3cd9d67bf49d01f6a2e33d0bd9bc804ddbe6ce1ff5d219c42624851db5dbc:1000:192.168.0.0/16       # changeme!

          3. Bounce the server to make sure you did that right!  Log back into GUI after bounce
             cd /home/tech/maltrail
             pkill -f server.py
             python server.py &

        • Wile editing config, add a line to move Maltrail log files out of /var/log to avoid issues associated with DietPi Ramlog  option (cleared maltrail logs)

          1. Modify maltrail.config per following:

# Directory used for log storage 
# TPed was here.  Moved the logs to maltrail directory so we don't bump
# heads with DietPi-RAMlog options (clears /var/log) 
# LOG_DIR $SYSTEM_LOG_DIR/maltrail
LOG_DIR /home/dietpi/maltrail/log 

2. • 1. • 2. Add a little script to /etc/cron.monthly directory to delete logs older than 1 year (season to your own taste). 
             // Mine is called maltrail_log_cleanup.sh, I also "chmod -x'd it ... not sure if that was necessary  )

#!/bin/bash

# Delete Maltrail log files older than 1 year

find /home/dietpi/maltrail/log -mtime +365 -delete

2. • 2. REMEMBER!  This is currently running in a terminal session!  Not as a service, DON'T EXIT SSH OR CLOSE TERMINAL SESSION!

// See "Pull it all Together" section below - We will turn this into a Service that starts when the system boots //

Sing-a-long with https://www.technicallywizardry.com/raspberry-pi-network-monitor/ (read carefully)
Command line syntax & more in https://github.com/zaneclaes/network-traffic-metrics#readme

1. Installs (some of this may have been already installed with Maltrail, but we'll keep components independent) 

cd /home/dietpi
sudo apt-get install tcpdump
sudo pip3 install argparse prometheus_client

git clone https://github.com/zaneclaes/network-traffic-metrics.git

cd ./network-traffic-metrics

2. I built a script to fire up eth0 monitor.  Season tcpdump filters to taste (192.168.x.0/24) and append with a "&" to run in background ...
   sudo python3 ./network-traffic-metrics.py -i eth0 -p 8001 "(src net 192.168.x.0/24 and not dst net 192.168.x.0/24) or (dst net 192.168.x.0/24 and not src net 192.168.x.0/24)" 

3. Test via browser @ given ports.  You should see statistics
   http://<ip>:8001

Install Prometheus with help of https://prometheus.io/docs/prometheus/latest/getting_started/

1. cd  /home/dietpi (don't install this under maltrail or network-monitor! I confused myself)

2. Download w/ help from https://pimylifeup.com/raspberry-pi-prometheus/

   • Check latest version (arm7 in my case) @ https://prometheus.io/download/

   • Insert version number and download via
     $ wget https://github.com/prometheus/prometheus/releases/download/v2.37.5/prometheus-2.37.5.linux-armv7.tar.gz

   • Unzip (also rename directory in hopes of simplification)
     tar xvfz prometheus-*.tar.gz
     mv prometheus-2.x.x.linux-armv7/ prometheus/

3. Modify  ./network-traffic-metrics/prometheus/prometheus.yml ... add localhost targets.  Now looks like this ...
   global:
   scrape_interval:     15s # How frequently to report
   external_labels:
     monitor: 'network-traffic-metrics'
   scrape_configs:
   - job_name: 'network-traffic-metrics'
   # Tped was here.  target changed to localhost and target of 8001
     static_configs:
       - targets: ['localhost:8001'] # The Network Traffic Metrics IP/port

4. Test it out
   ./prometheus --config.file=prometheus.yml

Then surf to metrics via:
http://<ip>:9090/metrics 

This step is now optional.  We have another RPI on the home network running Grafana so I decided to use that for the Network Traffic Monitor dashboard - takes a little more load off of the RPI2.

If needed, SentinelPi can easily run Grafana ... it is included in DietPi optimized software - setup is easy:

1. To install Grafana via Dietpi (runs as a service)
   run dietpi-software and select/install:
   77  Grafana: platform for analytics and monitoring 

• More info @ https://dietpi.com/docs/software/hardware_projects/#grafana

• Official docs @ https://grafana.com/docs/grafana/latest/installation/debian/.  

• Even better, a tutorial @ https://grafana.com/tutorials/install-grafana-on-raspberry-pi/

1. Surf to http://<ip>:3001 - login as admin/admin, wit will force PW change

2. Configure:  Add datasource and Network Traffic Monitor guys's dashboard.  

   • Click on gear in Grafana UI:

     1. Add Datasource->Choose Prometheus and set server to http://<ip of server>:9090

     2. Add Dashboard:  Clink on "+" then import Network Traffic Dashboard by number (12619)
        Once installed, set dashboard datasource to Prometheus (from step 1)

I'll want this contraption to come up by itself when the system boots.   Here is the current order of events ... 

1. Couple of crontab @reboot items (Append to file) ...
   sudo crontab -e

   • Put eth0 into promiscuous mode via crontab @reboot ...
     # TPed was here, put interfaces into promisc mode
     @reboot sudo ifconfig eth0 promisc

   • Network Traffic Monitor Auto Start:  Stick this in crontab @ reboot also ... per recommendation in doc.  Make sure python script is fully qualified!
     # TPed was here again!  Fire up network traffic monitor @ boot
     @reboot sudo python3 /home/pi/network-traffic-metrics/network-traffic-metrics.py -i eth0 -p 8001 "(src net 192.168.xx.0/24 and not dst net 192.168.xx.0/24) or (dst net 192.168.xx.0/24 and not src net 192.168.xx.0/24)" &

2. Maltrail:  Set to run as services so it auto starts after boot  

   • sensor.py:  This will run as a service.  "maltrail-sensor.service" provided w/ install and I can take a hint!

     1. First copy the provided maltrail -sensor.service file to where all the other .service files live!
        sudo cp /home/pi/maltrail/maltrail-sensor.service /usr/lib/systemd/system/maltrail-sensor.service

     2. Next, edit the .service file to set working directory to where I installed this thing.  Now looks like this:
        [Unit]
        Description=Malicious traffic detection sensor https://github.com/stamparm/maltrail
        [Service]
        User=root
        WorkingDirectory=/home/dietpi/maltrail
        ExecStart=/usr/bin/python3 sensor.py
        KillMode=mixed
        [Install]
        WantedBy=multi-user.target

   • server.py:  Same bit for server, it will run as a service too -  "maltrail-server.service" provided w/ install.

     1. Copy the provided maltrail-server.service file to systemd services area...
        sudo cp /home/pi/maltrail/maltrail-server.service /usr/lib/systemd/system/maltrail-server.service

     2. Edit the .service file to set working directory for server.py:
        [Unit]
        Description=Malicious traffic detection server https://github.com/stamparm/maltrail
        [Service]
        User=root
        WorkingDirectory=/home/dietpi/maltrail
        ExecStart=/usr/bin/python3 server.py
        KillMode=mixed
        [Install]
        WantedBy=multi-user.target

3. Grafana:   If installed via dietpi-software, it will already be a servcie and managed 

4. Prometheus  Auto Start via https://sbcode.net/prometheus/prometheus-service/

   • This guy will be fired up from prometheus path so remember to have the correct prometheus.yml in the directory (/home/pi/prometheus/ in my case).   Original yml was created under network-monitor thingy

   • Create service file @ /user/lib/systemd/system/prometheous.service
     [Unit]
     Description=Prometheus System Monitoring and Alerting software. More @ https://prometheus.io
     [Service]
     User=root
     WorkingDirectory=/home/pi/prometheus
     ExecStart=/home/pi/prometheus/prometheus --config.file=prometheus.yml
     KillMode=mixed
     [Install]
     WantedBy=multi-user.target

5. Commit hand-built services to Dietpi memory so everything fires up @ boot and is controlled in one place

   • chmod service files ... I instinctively use 644 for these
     sudo chmod 644 /usr/lib/systemd/system/maltrail-server.service
     sudo chmod 644 /usr/lib/systemd/system/maltrail-sensor.service
     sudo chmod 644 /usr/lib/systemd/system/prometheus.service

   • run dietpi-services utility to add Maltrail server and sensor and Prometeous:
     // Use "Add Missing Service" option, pretty simple //

Re-Boot-A-Roo to check things out ....

Traffic Monitor data @ http://<ip>:8001
Prometheous should be @ http://<ip>:9090/metrics
Grafana (if installed) will be @ http://<ip>:3001 

08 February 2023 - Prometheus is still dying:  compaction failed, 

25 January 2023 - Prometheus has been going into the ditch, rolled back to previous version (v2.31.5)

Sentinel Pi has been struggling with stats ... I'm wondering if Prometheus v2.37.5 has some arm/memory issues ...

log shows:  prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
// I also saw some sort of memory "panic" in the journal but lost that //

...  anyhoo ... I went back to the version of Prometheus I've used for past few years - prometheus-2.31.1.linux-armv7

07 January 2023 ... Whoops!  Missing maltrail history!

1. It looks like maltrail and dietpi ramlog option may not work well together.  The DietPi Ramlog option appears to clears ALL log files under /var/log  on a regular basis - including maltrail's which also happen to be used for historical purposes - if you clink back in time using maltrail calendar, it uses /var/log/maltrail info. 
   Fix:  Moved log files under the maltrail directory, they should be small (unless you are constantly under attack!).  Will watch storage and rotate if necessary

   • Modify maltrail.config per following:

# Directory used for log storage 
# TPed was here.  Moved the logs to maltrail directory so we don't bump
# heads with DietPi-RAMlog options (clears /var/log) 
# LOG_DIR $SYSTEM_LOG_DIR/maltrail
LOG_DIR /home/dietpi/maltrail/log 

1. • Added a little script to /etc/cron.monthly (maltrail_log_cleanup.sh)

#!/bin/bash

# Delete Maltrail log files older than 1 year

find /home/dietpi/maltrail/log -mtime +365 -delete

03 Jan 2023 ... this thing has been running GREAT!  Time to tweak it and mess it up!

• Complete rebuild using dietpi and 16g sd .... and folded into recipe above.  Install is MUCH easier under DietPi

• Maltrail, Network Traffic monitor and prometheous are still installed by hand (we'll watch for docker for next round)

• Grafana is still in the recipe but not used on our SentinelPi, we have another Grafana server on the home net

• Now looking for web-based packet capture project, it would be nice to take a close look at traffic periodically

November 2022:  Little tweak/update

1. Swapped in unused RPI2.  It seems to be handling everything fine.   Needed RPI3B+ for another project 

2. Added script to watch over RPI2 throttling and such - https://github.com/tped/PiPower

3. updated/upgraded stuff  (fingers crossed):   sudo apt update && sudo apt upgrade
   // Seemed to take care of OS & Grafana, the rest was left alone //

4. Added RPIMonitor to monitor the monitor!  https://xavierberger.github.io/RPi-Monitor-docs/10_index.html

April 2022:  Not bothering w/hostname resolution in Network Traffic Monitor anymore, I'm used to the IP addresses now ... It's like watching digital rain!

Prometheus has been going into the ditch periodically - running our of memory?  or something?
looks like this...

SentinelPi prometheus[392]: ts=2022-04-20T03:43:07.200Z caller=head.go:803 level=info component=tsdb msg="Head GC completed" duration=19.600777ms

SentinelPi prometheus[392]: ts=2022-04-20T03:43:07.367Z caller=db.go:830 level=error component=tsdb msg="compaction failed" err="compact head: head memory truncate: truncate chunks.HeadReadWriter: mmap, size 134217728: cannot allocate memory"

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.264Z caller=compact.go:518 level=info component=tsdb msg="write block" mint=1650420006283 maxt=1650427200000 ulid=01G12NFH6X53RW9510QTTDGS9S duration=666.434323ms

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.295Z caller=head.go:803 level=info component=tsdb msg="Head GC completed" duration=22.080236ms

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.298Z caller=db.go:830 level=error component=

tsdb msg="compaction failed" err="compact head: head memory truncate: truncate chunks.HeadReadWriter: write dat

a/chunks_head/000545: file already closed"

....

yada yada

....

prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

SentinelPi systemd[1]: prometheus.service: Failed with result 'exit-code'.

Prometheus version:

./prometheus --version

prometheus, version 2.31.1 (branch: HEAD, revision: 411021ada9ab41095923b8d2df9365b632fd40c3)

 build user:       root@b013bc8edd0b

 build date:       20211105-20:27:14

 go version:       go1.17.3

 platform:         linux/arm

I see some chatter @ https://github.com/prometheus/prometheus/issues/7378

Updating everything

apt update
apt upgrade

Looks like there is also a prometheus update - 2.35 - via latest version (arm7) @ https://prometheus.io/download/.  I scanned release notes from 2.32 to 2.35 for memory leaks and such ... maaaaybeeee?!   I'll watch and update prometheous next month

ToDo:  put together a SentinelPi Update procedure or script

---------------------

March 2022:  Little setback.  The AP (from junk drawer in my workroom) seems to have given up the ghost - like a VERY HARD RESET whilst it was running!!!   Unrelated to SentinelPi software & such ... BUT VERY ODD!!  NetGear WRN2000 factory reset appears to have changed its Serial Number & Default password - they are now different from those printed on the router @ factory!  No $h1t?!?!!  I can get it working, but it's running @ about 1/2 speed w/ no apparent symptoms (WiFi or Ethernet errors, retransmits, et

February 2022:  A few more ToDo's

• Polished up this recipe, the whole contraption now starts when SentinelPi is booted

• Things came up MUCH better this time around - I booted most of the closet when I switched over to AP for WiFi things
  // I'm guessing there were some ARPs cached or something that caused some angst the first time //

• ToDo:  If/When I rebuild, it may be best to use DietPi for this one - better control over hardware and runtime world

• Netmonitor:  Still trying to figure out how to display hostnames rather than local IPs ... playing with -fqdn option - Not going to bother!  I'm used to IP's now

January 2022.   Project looks do-able ... a few hurdles and ToDo's and Fixed Stuff

1. Hostnames (as opposed to ip addresses) would be nice  ... in both maltrail and traffic monitor.  

2. Something is wrong (static route?) when running things via the AP ... e.g. HomePi SSH is broke!  Maybe more! OK Now

3. Could I run even more monitors & tools?  Wireshark-like thing?  IDS, like snort?  What else?

4. Make this thing restart-able.  It's a B1tch to restart if I accidentally close my ssh session!  Updating recipe as I go ... 

   • Prometheus & Gafana were simple - just run them as systemd services

   • Maltrail May still need some work ... there was a systemd service template for the sensor but ... server needs to fireup at some point, I presume AFTER the sensor (which may be slow?).  Firing up server.py in rc.local seems to be OK

   • Network-traffic-monitor agent/sensor thingies - created a couple of scripts for now, will need to be set up as services or cron @boot

5. Network configuration issues to deal with .... Promiscuous mode only works with THINGS that are wired ...

   • All Wired THINGS seems fine.  SentinelPi sits on a port that receives mirrored Ethernet traffic from the port that heads to the ISP Router (Semi-Smart Switch)

   • WiFi THINGS:  running second copy of agent /w different port to add in wifi traffic SEEMS to work ... BUT ... Promiscuous Mode only picks up broadcast traffic on our THINGS SSID ...  Couple of things to try here

     1. Need better understanding of WiFi Monitor Mode - what can a Pi do?  Do I need external adapters, etc

        • Good Article @ https://networkengineering.stackexchange.com/questions/3774/using-wireless-cards-in-promiscuous-mode/3787#3787

        • No Guts/No Glory - add USB WiFi Dongle (I have a drawer-full of EDIMAX ).  It looks like it's good to go for Monitor Mode BUT I still don't know if that would work

     2. SentinelPi MAY need to be the main AP for these IoT world ... I think this would be easy (our IoT is on isolated SSID), but may take a bit more RPI horsepower 

        • May be able to do RPi Monitor Mode tests on SuitiePi (RaspAP)

        • Investigate other probe-like configurations ... e.g. Netbook

     3. Separate/Commercial WiFi AP is probably the best answer here.  Proximity is a problem in our home, IoT devices seems to be spreading throughout the house. We already have areas with weak WiFi signals ... probably time for Mesh anyhoo

December 2021.  Completed initial search for tools for this project.   

Settled on maltrail + a couple of copies of Network Traffic Monitor python scripts running on a PI 3 I had laying around.   PI has eth0 *and* wlan0 in promiscuous mode and is connected to our 'things' subnet via wifi and ethernet.  The ethernet port is switch mirror of traffic destined for our ISP router and should pick up all traffic from wired 'things' on our network.  

Initial software install/setup was all in foreground - no services, cron jobs or anything.  I just left it all running in an ssh terminal session to see if it would all work together ... it seems to be!  I've been watching cpu, disk space + whatever else I find to monitor.  It seems to be working, I see traffic in the tools!   I'm actually surprised! 

--------- Odd's n ends from initial work ... I may still salvage something from here ---

Tested a bunch of stuff.   Attempting to make a little gizmoPi that watches over our quickly growing collection of IoT "things".

Redoo-a-Roo #2! Latest will be above this from now on!

---------

NTOPng - I forgot about this guy! It works OK, but pretty much becomes useless after a short period of time (10 mins) . It is nice, but not worth subscription price for our home needs.

Install - started @ https://packages.ntop.org/ and downloaded RPI via:

wget https://packages.ntop.org/RaspberryPI/apt-ntop_1.0.190416-469_all.deb

sudo dpkg -i apt-ntop_1.0.190416-469_all.deb

sudo apt-get install ntopng nprobe n2n

Moving on to installation guide @ https://www.ntop.org/guides/ntopng/what_is_ntopng.html#installing-on-linux

Install DOES permanently install ... ntopng will start when the system is rebooted an sits right on top of grafana (port 3000).

// I'll shut it down manually when working on maltrail/netmon stuff //

------

Maltrail - Sing-along-with https://computingforgeeks.com/setup-maltrail-malicious-traffic-detection-system-on-linux/

Maltrail Readme is VERY good too ... find it @ https://github.com/stamparm/maltrail/blob/master/README.md

Install Notes - rough for me? So I can retrace my steps

• DietPie bullseye for test ... installed stuff as root(?!)

  • Dietpi-config

    • Passwords/hostname ... yada yada

    • enabled both eth0 and wlan0 (wifi config country, etc)

• Used 32g sd - hopefully big enough

• had to install apt-utils (dietpi side effect)

• Also installed net-tools to get netstat and ifconfig

  • ifconfig'd wlan0 and eth0 to be promiscuous (ifconfig eth0 promisc)

  • netstat -i to check - see "P" in flg column

• Also installed python3 instead of 2 ... python3-pcappy, pip

• Install recap
  sudo apt-get install schedtool
  sudo apt-get install git python3-pcapy -y
  git clone https://github.com/stamparm/maltrail.git

• Couple of conf mods

  • HTTP_ADDRESS = IP to listen on for GUI (I used eth0 address)

  • Admin password - under USERS.
    To change: create a new one using echo -n '<Strong Password goes here>' | sha256sum | cut -d " " -f 1

  • Set DNS to google (per guide) - modified /etc/dhcpcd.conf

• Start via 

  • • 1.  server/sensor.py &

• to run test install dnsutils

• running from root/maltrail/maltrail/ ...

• ui @ http://<ip or FQDN>:8338

Test using mirrored IoT switch port & promisc wifi ... We'll see!?

// Seemed to run ok, but ... can't recall why I did a re-doo-a-roo but I did! //

----------- Part 2 - IoT traffic Monitor PLUS maltrail ---------

RaspOS lite as base, more network traffic tools then maltrail

1. download RaspOS lite and flash (I got bullsye)

2. Update - whoops "InRelease changed" warnings are annoying when 'official' release is used,
   sudo apt-get update --allow-releaseinfo-change
   sudo apt-get upgrade

3. Sing-a-long with https://www.technicallywizardry.com/raspberry-pi-network-monitor/ (read carefully)
   No huge struggles thru testing .... Help along the way:
   - https://danielmiessler.com/study/tcpdump/
   - Commandline flags @ https://github.com/zaneclaes/network-traffic-metrics#configuration

4. Test run - remember to check promisc (netstat -i) ... we'll need to make this stick before the dust settles

   • Explicitly set Port and Interface since I plan (hope) to run two of these - one on wifi, other on mirrored ethernet
     for WiFi:
     sudo python3 ./network-traffic-metrics.py "(src net 192.168.x.0/24 and not dst net 192.168.x.0/24) or (dst net 192.168.x.0/24 and not src net 192.168.x.0/24)" -i wlan0 -p 8000

   • Test via browser @ http://<IP>:8000/metrics

5. Install Prometheus - help @ https://prometheus.io/docs/prometheus/latest/getting_started/

   • Download w/ help from https://pimylifeup.com/raspberry-pi-prometheus/

     1. Check latest version (arm7) @ https://prometheus.io/download/

     2. Insert version number and Download via
        $ wget https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-armv7.tar.gz

     3. Unzip via (also rename directory in hopes of simplification)
        tar xvfz prometheus-*.tar.gz
        mv prometheus-2.x.x.linux-armv7/ prometheus/

   • Modify prometheus.yaml ... added external_lable to global and localhost:8000 target initially (run on same raspi)
     
     global:
     scrape_interval: 15s # How frequently to report
     # TPed was here: added external_labels
     external_labels:
     monitor: 'network-traffic-metrics'
     scrape_configs:
     - job_name: 'network-traffic-metrics'
     static_configs
     # TPed was here, change target to Network traffic Metrics on localhost for now
     # - targets: ["localhost:9090"]
     - targets: ["localhost:8000"]

   • Test by firing up prometheus:
     ./prometheus --config.file=prometheus.yml

Then surfing to metrics via:

http://<ip>:9090/metrics

6. Install Grafana - official docs @ https://grafana.com/docs/grafana/latest/installation/debian/. Even better, a tutorial @ https://grafana.com/tutorials/install-grafana-on-raspberry-pi/

   • From Wizardly page:
     sudo apt-get install -y apt-transport-https
     sudo apt-get install -y software-properties-common wget

   • wget in next step seemed to fail ... but I think it was just a warning: "apt-key Is Deprecated" message ... If it is more than a warning, this may help: https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-how-to-add.html:
     wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
     echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
     sudo apt-get update
     sudo apt-get

   • FINALLY Install grafana ... Some setup hints popped out during install:
     sudo apt-get install grafana
     ... yada ... yada ...
     Unpacking grafana (8.3.2) ...
     Setting up grafana (8.3.2) ...
     Adding system user `grafana' (UID 110) ...
     Adding new user `grafana' (UID 110) with group `grafana' ...
     Not creating home directory `/usr/share/grafana'.
     ### NOT starting on installation, please execute the following statements to configure grafana to start automatically
     using systemd
     sudo /bin/systemctl daemon-reload
     sudo /bin/systemctl enable grafana-server
     ### You can start grafana-server by executing
     sudo /bin/systemctl start grafana-server

   • Started grfana server - just like it told me to ...
     sudo /bin/systemctl start grafana-server.service

   • Surf to new grafana server via http://<IP or localhost>:3000, login as admin/admin and change PW

7. Now hook it all together - add prometheus as data source to grafana. All steps below because I haven't set anything up to automatically run yet...also need to remember where I put everything ....

netstat -i // Make sure interfaces are in promiscuous mode //

cd ./network-traffic-metrics

sudo python3 ./network-traffic-metrics.py "(src net 192.168.xx.0/24 and not dst net 192.168.xx.0/24) or

(dst net 192.168.xx.0/24 and not src net 192.168.xx.0/24)" -i wlan0 -p 8000 &

cd prometheus

./prometheus --config.file=prometheus.yml &

test python via http://<ip>:8000

test prometheus via http://<ip>:9090

8. ALMOST FINALLY - add Prometheus as datasource to grafana ... more @ https://grafana.com/docs/grafana/v7.5/datasources/prometheus/
   there was a panel for this *or* click gear on left and "datasource"
   Named the datasource Prometheus
   Changed HTTP to point to prometheus service @ http://<ip>:9090
   save & test

9. FINALLY! Added whats-his-name's dashboard (from original network-monitoring metrics) ... using +, Dashboard on left menu and add dashboard 12619

10. POOF! It works! I'll be damned!!

11. To Do's based on initial observations ...

    • For our world:

      1. To see ALL IoT THINGS, I'll have to mirror an ethernet port to catch wired THINGS.
         Will try to fire-up a second python metrics gatherer w/ different port to watch eth0 & add to prometheus as another target

         • Second python script seems fine - used a different port
           Started both with shell scripts - I want to see console output. Some [Skips] that I may want to filter out of tcpdump

         • Prometheus - second target: I just added second target to the prometheus.yaml file - because I didn't know any better!
           - targets: ["localhost:8000", "localhost:8080]

      2. TOO MANY IP's! Ill try a hosts file on the monitor/spypi to see if I can name the hosts in the display. Our IoT stuff is predictable
         Seems to be getting resolved at the python level, probably fine ... BUT, nothing shows in grafana when I send it resolved IP addresses
         soooo. in grafana dashboard, I replaced Regex in LocalIPs variable with: [A-Za-z0-9\.\-]{0,} (letters, numbers, dots & dashes)
         Previously it was: /^((127\.\d+\.)|(10\.\d+\.)|(172\.1[6-9]\.)|(172\.2[0-9]\.)|(172\.3[0-1]\.)|(192\.168\.)\d+\.\d+)$/
         (rfc1918 private addresses, I think)

      3. Decide if I will run all of this on a single RPI (including the maltrail thingy)

Bookmark collection

More info @ https://haxf4rall.com/2018/06/13/maltrail-malicious-traffic-detection-system/ (some popup email collector, tho)

InternetPi on top ... from https://github.com/geerlingguy/internet-pi

target tools @ https://sectools.org/ + https://sectools.org/tag/traffic-monitors/ (old)

Net Monitor w/ Prometheus @ https://www.technicallywizardry.com/raspberry-pi-network-monitor/

Static Domain servers @ https://www.thegeekpub.com/18336/change-the-raspberry-pi-dns-settings/

Delete stuff from Prometheus in https://www.shellhacks.com/prometheus-delete-time-series-metrics/

• Admin mode: run w/ --web.enable-admin-api

• curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}'

• kill -TERM <prometheus#>