Raspberry PI IoT Network Monitor

Initially running on RPi 3 that was sitting here doing nothing - will watch carefully. Hopefully 32g SD will be enough, will watch this too!

Using RaspOS Lite on this one, mostly due to python scripts that appear to have been developed in this environment.
Guide is @ https://www.raspberrypi.com/documentation/computers/os.html

1. Download and Flash RaspOS Lite (Bullseye!). Raspberry Pi Imager is now easiest.

2. Initial Boot w/ Keyboard & Monitor attached

3. sudo raspi-config

   • System Options (kinda nice - main config options here now):

     1. Wireless Lan: Set Country Code, select SSID/pw of local IoT Land

     2. Password: Chang pi's pw

     3. Set Hostname

     4. Boot/Autologin: Console Autologin

     5. Wait for Network @ Boot

   • Interface Options

     1. Enable SSH

   • Localisation Options

     1. Locale: en_US.utf-8 UTF-8

     2. Timezone

     3. Keyboard: US

     4. WLAN Country Code (prob already set)

   • Advanced

     1. Expand Filesystem (probably already done)

   • Update raspi-config tool

4. <reboot-a-roo!> and switch to SSH for the rest of this

5. Update/Upgrade ... yada yada

   • sudo apt update

   • sudo apt full-upgrade

6. Editor: Don't forget joe (or your favorite editor)

   • sudo apt-get install joe

7. Set Static IP addresses (v4) and google dns per Network-Traffic-Monitor guy's recommendation

interface eth0
static ip_address=192.168.xx.x/24
static routers=192.168.xx.x
static domain_name_servers=8.8.8.8 8.8.4.4

interface wlan0
static ip_address=192.168.xx.x/24
static routers=192.168.xx.x
static domain_name_servers=8.8.8.8 8.8.4.4

8. sudo crontabPut eth0 into promiscuous mode via crontab @reboot ...
   // not sure this is THE correct way, but we'll see //
   sudo crontab -e
   // Append to file //
   # TPed was here, put interfaces into promisc mode
   @reboot sudo ifconfig eth0 promisc

9. Finally turn off wireless stuff ... There's MORE than enough RF in our little network closet already!!
   // reminder: rfkill survives through a boot ... to undo this use "unblock" //
   sudo rfkill block wifi
   sudo rfkill block bluetooth

Sing-a-long with https://github.com/stamparm/maltrail/blob/master/README.md#quick-start. The Whole README is very good!

1. Install
   
   sudo apt-get install git python3 python3-dev python3-pip python-is-python3 libpcap-dev build-essential procps schedtool
   sudo apt-get install git python3-pcapy -y
   sudo apt-get install dnsutils
   sudo pip3 install pcapy-ng
   git clone --depth 1 https://github.com/stamparm/maltrail.git
   

2. Initial Testing ...

   • Check Promiscuous mode (you should see "P" in flags)

netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 151201 0 0 0 70372 0 0 0 BMPRU
lo 65536 14 0 0 0 14 0 0 0 LRU

2. • Fire up sensor ... it will take a bit as it updates trails, feeds or whatevers
     

   • Install and fire up maltrail server (gui)
     git clone --depth 1 https://github.com/stamparm/maltrail.git
     // I don't think this was needed //
     sudo python3 server.py &

   • Test via browser @ http://<ipaddress>:8338

     1. Initial login is admin/changeme!

     2. A couple of queries to get something logged (seem to be dangerous places! Test was in README)
        ping -c 1 136.161.101.53
        nslookup morphed.ru

     3. Refresh maltrail web interface to see logged threats

   • Complete Install and Auto-Start once we know this thing works!

     1. Minimal config file touch-ups - /home/pi/maltrail/mailtrail.conf

        • Change password:

          1. Create a new password like this:
             echo -n '<NewStrongPassword>' | sha256sum | cut -d " " -f 1 1ee0cd0713ba2b4fde13881618e0995ee2f560c41fb7b2d195847e5a018fc03c2

          2. Edit maltrail.conf and plant your new password (:0: at end) in the USERS section, replace the default password (changeme!)
             # User entries (username:sha256(password):UID:filter_netmask(s))
             # Note(s): sha256(password) can be generated on Linux with: echo -n 'password' | sha256sum | cut -d " " -1
             # UID >= 1000 have only rights to display results (Note: this moment only functionality implemented at the c
             # filter_netmask(s) is/are used to filter results
             USERS
             admin:1ee0cd0713ba2b4fde13881618e0995ee2f560c41fb7b2d195847e5a018fc03c2:0: # NewStrongPassword
             # local:9ab3cd9d67bf49d01f6a2e33d0bd9bc804ddbe6ce1ff5d219c42624851db5dbc:1000:192.168.0.0/16 # changeme!

          3. Bounce the server to make sure you did that right! Log back into GUI after bounce
             cd /home/tech/maltrail
             pkill -f server.py
             python server.py &

     2. REMEMBER! This is currently running in a terminal session! Not as a service, DON'T EXIT SSH OR CLOSE TERMINAL SESSION!

// See "Pull it all Together" section below - We will turn this into a Service that starts when the system boots //

Sing-a-long with https://www.technicallywizardry.com/raspberry-pi-network-monitor/ (read carefully)
Command line syntax & more in https://github.com/zaneclaes/network-traffic-metrics#readme

1. Installs (some of this was already installed with Maltrail, but we'll keep components independent)

cd /home/pi
sudo apt-get install git python3-pip tcpdump
sudo pip3 install argparse prometheus_client

git clone https://github.com/zaneclaes/network-traffic-metrics.git

cd ./network-traffic-metrics

2. A Script to fire up eth0 monitor. Season tcpdump filters to taste (192.168.x.0/24) and append with a "&" to run in background ... May decide to run these as services
   sudo python3 -i eth0 -p 8001 ./network-traffic-metrics.py "(src net 192.168.x.0/24 and not dst net 192.168.x.0/24) or (dst net 192.168.x.0/24 and not src net 192.168.x.0/24)"

3. Test via browser @ given ports. You should see statistics
   http://<ip>:8001

Install Prometheus with help of https://prometheus.io/docs/prometheus/latest/getting_started/

1. cd /home/pi (don't install this under maltrail or network-monitor!)

2. Download w/ help from https://pimylifeup.com/raspberry-pi-prometheus/

   • Check latest version (arm7) @ https://prometheus.io/download/

   • Insert version number and Download via
     $ wget https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-armv7.tar.gz

   • Unzip (also rename directory in hopes of simplification)
     tar xvfz prometheus-*.tar.gz
     mv prometheus-2.x.x.linux-armv7/ prometheus/

3. Modify ./network-traffic-metrics/prometheus/prometheus.yml ... add localhost targets. Now looks like this ...
   global:
   scrape_interval: 15s # How frequently to report
   external_labels:
   monitor: 'network-traffic-metrics'
   scrape_configs:
   - job_name: 'network-traffic-metrics'
   # Tped was here. target changed to localhost and target of 8001
   static_configs:
   - targets: ['localhost:8001'] # The Network Traffic Metrics IP/port

4. Test it out
   ./prometheus --config.file=prometheus.yml

Then surf to metrics via:
http://<ip>:9090/metrics

Install Grafana - official docs @ https://grafana.com/docs/grafana/latest/installation/debian/.
Even better, a tutorial @ https://grafana.com/tutorials/install-grafana-on-raspberry-pi/

1. From Wizardly page:
   sudo apt-get install -y apt-transport-https
   sudo apt-get install -y software-properties-common wget

   • wget in next step seemed to fail ... but I think it was just a warning: "apt-key Is Deprecated" message ... If it is more than a warning, this may help: https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-how-to-add.html:
     wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
     echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
     sudo apt-get update

   • FINALLY Install grafana ... Some setup hints popped out during install:
     sudo apt-get install grafana
     ... yada ... yada ...
     Unpacking grafana (8.3.2) ...
     Setting up grafana (8.3.2) ...
     Adding system user `grafana' (UID 110) ...
     Adding new user `grafana' (UID 110) with group `grafana' ...
     Not creating home directory `/usr/share/grafana'.
     ### NOT starting on installation, please execute the following statements to configure grafana to start automatically
     using systemd
     sudo /bin/systemctl daemon-reload
     sudo /bin/systemctl enable grafana-server
     ### You can start grafana-server by executing
     sudo /bin/systemctl start grafana-server

2. Started grfana server - just like it told me to ...
   sudo /bin/systemctl start grafana-server.service

3. Surf to http://<ip>:3000 - login as admin/admin, wit will force PW change

4. Configure: Add datasource and Network Traffic Monitor guys's dashboard.
   Click on gear in Grafana UI:

   • Add Datasource->Choose Prometheus and set server to http://<ip of server>:9090

   • Add Dashboard: Clink on "+" then import Network Traffic Dashboard by number (12619)
     set datasource to prometheus

I'll want this contraption to come up by itself when the system boots.
(this may change ... seems clumsy, but it works. Will probably incorporate final AutoStart into above setup/config)

Here is the current order of events ...

1. Put eth0 into promiscuous mode via crontab @reboot ... // This seems to work //
   sudo crontab -e
   // Append to file //
   # TPed was here, put interfaces into promisc mode
   @reboot sudo ifconfig eth0 promisc

2. Network Traffic Monitor Auto Start: Stick this in crontab @ reboot also ... per recommendation in doc:
   (make sure python script is fully qualified!)
   # TPed was here again! Fire up network traffic monitor @ boot
   @reboot sudo python3 /home/pi/network-traffic-metrics/network-traffic-metrics.py -i eth0 -p 8001 "(src net 192.168.xx.0/24 and not dst net 192.168.xx.0/24) or (dst net 192.168.xx.0/24 and not src net 192.168.xx.0/24)" &

3. Maltrail Auto Start after boot // This seems solid //

   • sensor.py: This will run as a service. "maltrail.service" provided w/ install and I can take a hint!

     1. First copy the provided maltrail .service file to where all the other .service files live! (I checked grafana install)
        sudo cp /home/pi/maltrail/maltrail.service /usr/lib/systemd/system/maltrail.service

     2. Next, Edit the .service file to set working directory to where I installed this thing. Now looks like this:
        [Unit]
        Description=Malicious traffic detection system https://github.com/stamparm/maltrail
        [Service]
        User=root
        WorkingDirectory=/home/pi/maltrail
        ExecStart=/usr/bin/python3 sensor.py
        KillMode=mixed
        [Install]
        WantedBy=multi-user.target

     3. pretty sure we need to chmod that dang thang ....
        sudo chmod 644 /usr/lib/systemd/system/maltrail.service

     4. Check our work ....

        1. Kill whatever we have running ....
           sudo pkill -f server.py
           sudo pkill -f sensor.py

        2. Now clear the deck and start the the sensor as a service
           sudo systemctl daemon-reload
           sudo systemctl enable maltrail.service
           sudo systemctl start maltrail.service

   • server.py: Life seems to be good if we fire this up using rc.local so it starts after services. // seems to work fine //

     1. Edit /etc/rc.local and append ... server using absolute path to be safe. Bottom of file now looks like this ...
        # By default this script does nothing.
        # Print the IP address
        _IP=$(hostname -I) || true
        if [ "$_IP" ]; then
        printf "My IP address is %s\n" "$_IP"
        fi
        # TPed was here. Fire up maltrail server (I'd think sensor.service is already running)
        python3 /home/pi/maltrail/server.py &
        exit 0
        

4. Grafana: Was originally installed as an Auto Start service

5. Prometheus Auto Start via https://sbcode.net/prometheus/prometheus-service/

   • This guy will be fired up from prometheus path so remember to have the correct prometheus.yml in the directory (/home/pi/prometheus/ in my case). Original yml was created under network-monitor thingy

   • Create service file @ /user/lib/systemd/system/prometheous.service
     [Unit]
     Description=Prometheus System Monitoring and Alerting software. More @ https://prometheus.io
     [Service]
     User=root
     WorkingDirectory=/home/pi/prometheus
     ExecStart=/home/pi/prometheus/prometheus --config.file=prometheus.yml
     KillMode=mixed
     [Install]
     WantedBy=multi-user.target

   • This should commit it all to memory for next boot *and* fire it up now
     sudo systemctl daemon-reload
     sudo systemctl enable prometheus.service
     sudo systemctl start prometheus.service

November 2022: Little tweak/update

1. Swapped in unused RPI2. It seems to be handling everything fine. Needed RPI3B+ for another project

2. Added script to watch over RPI2 throttling and such - https://github.com/tped/PiPower

3. updated/upgraded stuff (fingers crossed): sudo apt update && sudo apt upgrade
   // Seemed to take care of OS & Grafana, the rest was left alone //

4. Added RPIMonitor to monitor the monitor! https://xavierberger.github.io/RPi-Monitor-docs/10_index.html

April 2022: Not bothering w/hostname resolution in Network Traffic Monitor anymore, I'm used to the IP addresses now ... It's like watching digital rain!

Prometheus has been going into the ditch periodically - running our of memory? or something?
looks like this...

SentinelPi prometheus[392]: ts=2022-04-20T03:43:07.200Z caller=head.go:803 level=info component=tsdb msg="Head GC completed" duration=19.600777ms

SentinelPi prometheus[392]: ts=2022-04-20T03:43:07.367Z caller=db.go:830 level=error component=tsdb msg="compaction failed" err="compact head: head memory truncate: truncate chunks.HeadReadWriter: mmap, size 134217728: cannot allocate memory"

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.264Z caller=compact.go:518 level=info component=tsdb msg="write block" mint=1650420006283 maxt=1650427200000 ulid=01G12NFH6X53RW9510QTTDGS9S duration=666.434323ms

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.295Z caller=head.go:803 level=info component=tsdb msg="Head GC completed" duration=22.080236ms

SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.298Z caller=db.go:830 level=error component=

tsdb msg="compaction failed" err="compact head: head memory truncate: truncate chunks.HeadReadWriter: write dat

a/chunks_head/000545: file already closed"

....

yada yada

....

prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

SentinelPi systemd[1]: prometheus.service: Failed with result 'exit-code'.

Prometheus version:

./prometheus --version

prometheus, version 2.31.1 (branch: HEAD, revision: 411021ada9ab41095923b8d2df9365b632fd40c3)

build user: root@b013bc8edd0b

build date: 20211105-20:27:14

go version: go1.17.3

platform: linux/arm

I see some chatter @ https://github.com/prometheus/prometheus/issues/7378

Updating everything

apt update
apt upgrade

Looks like there is also a prometheus update - 2.35 - via latest version (arm7) @ https://prometheus.io/download/. I scanned release notes from 2.32 to 2.35 for memory leaks and such ... maaaaybeeee?! I'll watch and update prometheous next month

ToDo: put together a SentinelPi Update procedure or script

---------------------

March 2022: Little setback. The AP (from junk drawer in my workroom) seems to have given up the ghost - like a VERY HARD RESET whilst it was running!!! Unrelated to SentinelPi software & such ... BUT VERY ODD!! NetGear WRN2000 factory reset appears to have changed its Serial Number & Default password - they are now different from those printed on the router @ factory! No $h1t?!?!! I can get it working, but it's running @ about 1/2 speed w/ no apparent symptoms (WiFi or Ethernet errors, retransmits, et

February 2022: A few more ToDo's

• Polished up this recipe, the whole contraption now starts when SentinelPi is booted

• Things came up MUCH better this time around - I booted most of the closet when I switched over to AP for WiFi things
  // I'm guessing there were some ARPs cached or something that caused some angst the first time //

• ToDo: If/When I rebuild, it may be best to use DietPi for this one - better control over hardware and runtime world

• Netmonitor: Still trying to figure out how to display hostnames rather than local IPs ... playing with -fqdn option - Not going to bother! I'm used to IP's now

January 2022. Project looks do-able ... a few hurdles and ToDo's and Fixed Stuff

1. Hostnames (as opposed to ip addresses) would be nice ... in both maltrail and traffic monitor.

2. Something is wrong (static route?) when running things via the AP ... e.g. HomePi SSH is broke! Maybe more! OK Now

3. Could I run even more monitors & tools? Wireshark-like thing? IDS, like snort? What else?

4. Make this thing restart-able. It's a B1tch to restart if I accidentally close my ssh session! Updating recipe as I go ...

   • Prometheus & Gafana were simple - just run them as systemd services

   • Maltrail May still need some work ... there was a systemd service template for the sensor but ... server needs to fireup at some point, I presume AFTER the sensor (which may be slow?). Firing up server.py in rc.local seems to be OK

   • Network-traffic-monitor agent/sensor thingies - created a couple of scripts for now, will need to be set up as services or cron @boot

5. Network configuration issues to deal with .... Promiscuous mode only works with THINGS that are wired ...

   • All Wired THINGS seems fine. SentinelPi sits on a port that receives mirrored Ethernet traffic from the port that heads to the ISP Router (Semi-Smart Switch)

   • WiFi THINGS: running second copy of agent /w different port to add in wifi traffic SEEMS to work ... BUT ... Promiscuous Mode only picks up broadcast traffic on our THINGS SSID ... Couple of things to try here

     1. Need better understanding of WiFi Monitor Mode - what can a Pi do? Do I need external adapters, etc

        • Good Article @ https://networkengineering.stackexchange.com/questions/3774/using-wireless-cards-in-promiscuous-mode/3787#3787

        • No Guts/No Glory - add USB WiFi Dongle (I have a drawer-full of EDIMAX ). It looks like it's good to go for Monitor Mode BUT I still don't know if that would work

     2. SentinelPi MAY need to be the main AP for these IoT world ... I think this would be easy (our IoT is on isolated SSID), but may take a bit more RPI horsepower

        • May be able to do RPi Monitor Mode tests on SuitiePi (RaspAP)

        • Investigate other probe-like configurations ... e.g. Netbook

     3. Separate/Commercial WiFi AP is probably the best answer here. Proximity is a problem in our home, IoT devices seems to be spreading throughout the house. We already have areas with weak WiFi signals ... probably time for Mesh anyhoo

December 2021. Completed initial search for tools for this project.

Settled on maltrail + a couple of copies of Network Traffic Monitor python scripts running on a PI 3 I had laying around. PI has eth0 *and* wlan0 in promiscuous mode and is connected to our 'things' subnet via wifi and ethernet. The ethernet port is switch mirror of traffic destined for our ISP router and should pick up all traffic from wired 'things' on our network.

Initial software install/setup was all in foreground - no services, cron jobs or anything. I just left it all running in an ssh terminal session to see if it would all work together ... it seems to be! I've been watching cpu, disk space + whatever else I find to monitor. It seems to be working, I see traffic in the tools! I'm actually surprised!

--------- Odd's n ends from initial work ... I may still salvage something from here ---

Tested a bunch of stuff. Attempting to make a little gizmoPi that watches over our quickly growing collection of IoT "things".

Redoo-a-Roo #2! Latest will be above this from now on!

---------

NTOPng - I forgot about this guy! It works OK, but pretty much becomes useless after a short period of time (10 mins) . It is nice, but not worth subscription price for our home needs.

Install - started @ https://packages.ntop.org/ and downloaded RPI via:

wget https://packages.ntop.org/RaspberryPI/apt-ntop_1.0.190416-469_all.deb

sudo dpkg -i apt-ntop_1.0.190416-469_all.deb

sudo apt-get install ntopng nprobe n2n

Moving on to installation guide @ https://www.ntop.org/guides/ntopng/what_is_ntopng.html#installing-on-linux

Install DOES permanently install ... ntopng will start when the system is rebooted an sits right on top of grafana (port 3000).

// I'll shut it down manually when working on maltrail/netmon stuff //

------

Maltrail - Sing-along-with https://computingforgeeks.com/setup-maltrail-malicious-traffic-detection-system-on-linux/

Maltrail Readme is VERY good too ... find it @ https://github.com/stamparm/maltrail/blob/master/README.md

Install Notes - rough for me? So I can retrace my steps

• DietPie bullseye for test ... installed stuff as root(?!)

  • Dietpi-config

    • Passwords/hostname ... yada yada

    • enabled both eth0 and wlan0 (wifi config country, etc)

• Used 32g sd - hopefully big enough

• had to install apt-utils (dietpi side effect)

• Also installed net-tools to get netstat and ifconfig

  • ifconfig'd wlan0 and eth0 to be promiscuous (ifconfig eth0 promisc)

  • netstat -i to check - see "P" in flg column

• Also installed python3 instead of 2 ... python3-pcappy, pip

• Install recap
  sudo apt-get install schedtool
  sudo apt-get install git python3-pcapy -y
  git clone https://github.com/stamparm/maltrail.git

• Couple of conf mods

  • HTTP_ADDRESS = IP to listen on for GUI (I used eth0 address)

  • Admin password - under USERS.
    To change: create a new one using echo -n '<Strong Password goes here>' | sha256sum | cut -d " " -f 1

  • Set DNS to google (per guide) - modified /etc/dhcpcd.conf

• Start via

  • • 1. server/sensor.py &

• to run test install dnsutils

• running from root/maltrail/maltrail/ ...

• ui @ http://<ip or FQDN>:8338

Test using mirrored IoT switch port & promisc wifi ... We'll see!?

// Seemed to run ok, but ... can't recall why I did a re-doo-a-roo but I did! //

----------- Part 2 - IoT traffic Monitor PLUS maltrail ---------

RaspOS lite as base, more network traffic tools then maltrail

1. download RaspOS lite and flash (I got bullsye)

2. Update - whoops "InRelease changed" warnings are annoying when 'official' release is used,
   sudo apt-get update --allow-releaseinfo-change
   sudo apt-get upgrade

3. Sing-a-long with https://www.technicallywizardry.com/raspberry-pi-network-monitor/ (read carefully)
   No huge struggles thru testing .... Help along the way:
   - https://danielmiessler.com/study/tcpdump/
   - Commandline flags @ https://github.com/zaneclaes/network-traffic-metrics#configuration

4. Test run - remember to check promisc (netstat -i) ... we'll need to make this stick before the dust settles

   • Explicitly set Port and Interface since I plan (hope) to run two of these - one on wifi, other on mirrored ethernet
     for WiFi:
     sudo python3 ./network-traffic-metrics.py "(src net 192.168.x.0/24 and not dst net 192.168.x.0/24) or (dst net 192.168.x.0/24 and not src net 192.168.x.0/24)" -i wlan0 -p 8000

   • Test via browser @ http://<IP>:8000/metrics

5. Install Prometheus - help @ https://prometheus.io/docs/prometheus/latest/getting_started/

   • Download w/ help from https://pimylifeup.com/raspberry-pi-prometheus/

     1. Check latest version (arm7) @ https://prometheus.io/download/

     2. Insert version number and Download via
        $ wget https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-armv7.tar.gz

     3. Unzip via (also rename directory in hopes of simplification)
        tar xvfz prometheus-*.tar.gz
        mv prometheus-2.x.x.linux-armv7/ prometheus/

   • Modify prometheus.yaml ... added external_lable to global and localhost:8000 target initially (run on same raspi)
     
     global:
     scrape_interval: 15s # How frequently to report
     # TPed was here: added external_labels
     external_labels:
     monitor: 'network-traffic-metrics'
     scrape_configs:
     - job_name: 'network-traffic-metrics'
     static_configs
     # TPed was here, change target to Network traffic Metrics on localhost for now
     # - targets: ["localhost:9090"]
     - targets: ["localhost:8000"]

   • Test by firing up prometheus:
     ./prometheus --config.file=prometheus.yml

Then surfing to metrics via:

http://<ip>:9090/metrics

6. Install Grafana - official docs @ https://grafana.com/docs/grafana/latest/installation/debian/. Even better, a tutorial @ https://grafana.com/tutorials/install-grafana-on-raspberry-pi/

   • From Wizardly page:
     sudo apt-get install -y apt-transport-https
     sudo apt-get install -y software-properties-common wget

   • wget in next step seemed to fail ... but I think it was just a warning: "apt-key Is Deprecated" message ... If it is more than a warning, this may help: https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-how-to-add.html:
     wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
     echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
     sudo apt-get update
     sudo apt-get

   • FINALLY Install grafana ... Some setup hints popped out during install:
     sudo apt-get install grafana
     ... yada ... yada ...
     Unpacking grafana (8.3.2) ...
     Setting up grafana (8.3.2) ...
     Adding system user `grafana' (UID 110) ...
     Adding new user `grafana' (UID 110) with group `grafana' ...
     Not creating home directory `/usr/share/grafana'.
     ### NOT starting on installation, please execute the following statements to configure grafana to start automatically
     using systemd
     sudo /bin/systemctl daemon-reload
     sudo /bin/systemctl enable grafana-server
     ### You can start grafana-server by executing
     sudo /bin/systemctl start grafana-server

   • Started grfana server - just like it told me to ...
     sudo /bin/systemctl start grafana-server.service

   • Surf to new grafana server via http://<IP or localhost>:3000, login as admin/admin and change PW

7. Now hook it all together - add prometheus as data source to grafana. All steps below because I haven't set anything up to automatically run yet...also need to remember where I put everything ....

netstat -i // Make sure interfaces are in promiscuous mode //

cd ./network-traffic-metrics

sudo python3 ./network-traffic-metrics.py "(src net 192.168.xx.0/24 and not dst net 192.168.xx.0/24) or

(dst net 192.168.xx.0/24 and not src net 192.168.xx.0/24)" -i wlan0 -p 8000 &

cd prometheus

./prometheus --config.file=prometheus.yml &

test python via http://<ip>:8000

test prometheus via http://<ip>:9090

8. ALMOST FINALLY - add Prometheus as datasource to grafana ... more @ https://grafana.com/docs/grafana/v7.5/datasources/prometheus/
   there was a panel for this *or* click gear on left and "datasource"
   Named the datasource Prometheus
   Changed HTTP to point to prometheus service @ http://<ip>:9090
   save & test

9. FINALLY! Added whats-his-name's dashboard (from original network-monitoring metrics) ... using +, Dashboard on left menu and add dashboard 12619

10. POOF! It works! I'll be damned!!

11. To Do's based on initial observations ...

    • For our world:

      1. To see ALL IoT THINGS, I'll have to mirror an ethernet port to catch wired THINGS.
         Will try to fire-up a second python metrics gatherer w/ different port to watch eth0 & add to prometheus as another target

         • Second python script seems fine - used a different port
           Started both with shell scripts - I want to see console output. Some [Skips] that I may want to filter out of tcpdump

         • Prometheus - second target: I just added second target to the prometheus.yaml file - because I didn't know any better!
           - targets: ["localhost:8000", "localhost:8080]

      2. TOO MANY IP's! Ill try a hosts file on the monitor/spypi to see if I can name the hosts in the display. Our IoT stuff is predictable
         Seems to be getting resolved at the python level, probably fine ... BUT, nothing shows in grafana when I send it resolved IP addresses
         soooo. in grafana dashboard, I replaced Regex in LocalIPs variable with: [A-Za-z0-9\.\-]{0,} (letters, numbers, dots & dashes)
         Previously it was: /^((127\.\d+\.)|(10\.\d+\.)|(172\.1[6-9]\.)|(172\.2[0-9]\.)|(172\.3[0-1]\.)|(192\.168\.)\d+\.\d+)$/
         (rfc1918 private addresses, I think)

      3. Decide if I will run all of this on a single RPI (including the maltrail thingy)

Bookmark collection

More info @ https://haxf4rall.com/2018/06/13/maltrail-malicious-traffic-detection-system/ (some popup email collector, tho)

InternetPi on top ... from https://github.com/geerlingguy/internet-pi

target tools @ https://sectools.org/ + https://sectools.org/tag/traffic-monitors/ (old)

Net Monitor w/ Prometheus @ https://www.technicallywizardry.com/raspberry-pi-network-monitor/

Static Domain servers @ https://www.thegeekpub.com/18336/change-the-raspberry-pi-dns-settings/

Delete stuff from Prometheus in https://www.shellhacks.com/prometheus-delete-time-series-metrics/

• Admin mode: run w/ --web.enable-admin-api

• curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}'

• kill -TERM <prometheus#>