Raspberry PI IoT Network Monitor

Sing-a-long with https://github.com/stamparm/maltrail/blob/master/README.md#quick-start.  The Whole README is very good!

1. Couple of leftover installs for this guy (some may already be there)
   
   sudo apt-get install python-is-python3 libpcap-dev schedtool python3-pcapy
   sudo pip3 install pcapy-ng
   git clone --depth 1 https://github.com/stamparm/maltrail.git
   
   

2. Initial Testing ...

   • Set/Check Promiscuous mode on Ethernet Adapter (you should see "P" in flags)
     // This will not live through a reboot ... we'll add it to cron @boot ultimately //
     
     To set: 
     ifconfig eth0 promisc
     
     To check: 

netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0      1500   151201      0      0 0         70372      0      0      0 BMPRU
lo       65536       14      0      0 0            14      0      0      0 LRU

2. • Fire up sensor ... it will take a bit as it updates trails, feeds or whatevers
     cd maltrail
     sudo python3 sensor.py &

   • Install and fire up maltrail server (gui)
     sudo python3 server.py &

   • Test via browser @ http://<ipaddress>:8338

     1. Initial login is admin/changeme!

     2. A couple of queries to get something logged (seem to be dangerous places!  Test was in README)
        ping -c 1 136.161.101.53  
        nslookup morphed.ru

     3. Refresh maltrail web interface to see logged threats

   • Complete Install and Auto-Start once we know this thing works!

     1. Minimal config file touch-ups - /home/pi/maltrail/mailtrail.conf 

        • Change password:

          1. Create a new password like this:
             echo -n '<NewStrongPassword>' | sha256sum | cut -d " " -f 1 

          2. Edit maltrail.conf and plant your new password (:0: at end) in the USERS section, replace the default password (changeme!)
             # User entries (username:sha256(password):UID:filter_netmask(s))
             # Note(s): sha256(password) can be generated on Linux with: echo -n 'password' | sha256sum | cut -d " " -1
             #          UID >= 1000 have only rights to display results (Note: this moment only functionality implemented at the c
             #          filter_netmask(s) is/are used to filter results
             USERS
               admin:1ee0cd0713ba2b4fde13881618e0995ee2f560c41fb7b2d195847e5a018fc03c2:0:                  # NewStrongPassword
             #   local:9ab3cd9d67bf49d01f6a2e33d0bd9bc804ddbe6ce1ff5d219c42624851db5dbc:1000:192.168.0.0/16       # changeme!

          3. Bounce the server to make sure you did that right!  Log back into GUI after bounce
             cd /home/tech/maltrail
             pkill -f server.py
             python server.py &

        • Wile editing config, add a line to move Maltrail log files out of /var/log to avoid issues associated with DietPi Ramlog  option (cleared maltrail logs)

          1. Modify maltrail.config per following:

# Directory used for log storage 
# TPed was here.  Moved the logs to maltrail directory so we don't bump
# heads with DietPi-RAMlog options (clears /var/log) 
# LOG_DIR $SYSTEM_LOG_DIR/maltrail
LOG_DIR /home/dietpi/maltrail/log 

2. • 1. • 2. Add a little script to /etc/cron.monthly directory to delete logs older than 1 year (season to your own taste). 
             // Mine is called maltrail_log_cleanup.sh, I also "chmod -x'd it ... not sure if that was necessary  )

#!/bin/bash

# Delete Maltrail log files older than 1 year

find /home/dietpi/maltrail/log -mtime +365 -delete

2. • 2. REMEMBER!  This is currently running in a terminal session!  Not as a service, DON'T EXIT SSH OR CLOSE TERMINAL SESSION!

// See "Pull it all Together" section below - We will turn this into a Service that starts when the system boots //

Install Prometheus with help of https://prometheus.io/docs/prometheus/latest/getting_started/

1. cd  /home/dietpi (don't install this under maltrail or network-monitor! I confused myself)

2. Download w/ help from https://pimylifeup.com/raspberry-pi-prometheus/

   • Check latest version (arm7 in my case) @ https://prometheus.io/download/

   • Insert version number and download via
     $ wget https://github.com/prometheus/prometheus/releases/download/v2.37.5/prometheus-2.37.5.linux-armv7.tar.gz

   • Unzip (also rename directory in hopes of simplification)
     tar xvfz prometheus-*.tar.gz
     mv prometheus-2.x.x.linux-armv7/ prometheus/

3. Modify  ./network-traffic-metrics/prometheus/prometheus.yml ... add localhost targets.  Now looks like this ...
   global:
   scrape_interval:     15s # How frequently to report
   external_labels:
     monitor: 'network-traffic-metrics'
   scrape_configs:
   - job_name: 'network-traffic-metrics'
   # Tped was here.  target changed to localhost and target of 8001
     static_configs:
       - targets: ['localhost:8001'] # The Network Traffic Metrics IP/port

4. Test it out
   ./prometheus --config.file=prometheus.yml

Then surf to metrics via:
http://<ip>:9090/metrics 

I'll want this contraption to come up by itself when the system boots.   Here is the current order of events ... 

1. Couple of crontab @reboot items (Append to file) ...
   crontab -e

   • Put eth0 into promiscuous mode via crontab @reboot ...
     # TPed was here, put interfaces into promisc mode
     @reboot sudo ifconfig eth0 promisc

   • Network Traffic Monitor Auto Start:  Stick this in crontab @ reboot also ... per recommendation in doc.  Make sure python script is fully qualified!
     # TPed was here again!  Fire up network traffic monitor @ boot
     @reboot sudo python3 /home/pi/network-traffic-metrics/network-traffic-metrics.py -i eth0 -p 8001 "(src net 192.168.xx.0/24 and not dst net 192.168.xx.0/24) or (dst net 192.168.xx.0/24 and not src net 192.168.xx.0/24)" &

2. Maltrail:  Set to run as services so it auto starts after boot  

   • sensor.py:  This will run as a service.  "maltrail-sensor.service" provided w/ install and I can take a hint!

     1. First copy the provided maltrail -sensor.service file to where all the other .service files live!
        sudo cp /home/pi/maltrail/maltrail-sensor.service /usr/lib/systemd/system/maltrail-sensor.service

     2. Next, edit the .service file to set working directory to where I installed this thing.  Now looks like this:
        [Unit]
        Description=Malicious traffic detection sensor https://github.com/stamparm/maltrail
        [Service]
        User=root
        WorkingDirectory=/home/dietpi/maltrail
        ExecStart=/usr/bin/python3 sensor.py
        KillMode=mixed
        [Install]
        WantedBy=multi-user.target

   • server.py:  Same bit for server, it will run as a service too -  "maltrail-server.service" provided w/ install.

     1. Copy the provided maltrail-server.service file to systemd services area...
        sudo cp /home/pi/maltrail/maltrail-server.service /usr/lib/systemd/system/maltrail-server.service

     2. Edit the .service file to set working directory for server.py:
        [Unit]
        Description=Malicious traffic detection server https://github.com/stamparm/maltrail
        [Service]
        User=root
        WorkingDirectory=/home/dietpi/maltrail
        ExecStart=/usr/bin/python3 server.py
        KillMode=mixed
        [Install]
        WantedBy=multi-user.target

3. Grafana:   If installed via dietpi-software, it will already be a servcie and managed 

4. Prometheus  Auto Start via https://sbcode.net/prometheus/prometheus-service/

   • This guy will be fired up from prometheus path so remember to have the correct prometheus.yml in the directory (/home/pi/prometheus/ in my case).   Original yml was created under network-monitor thingy

   • Create service file @ /user/lib/systemd/system/prometheous.service
     [Unit]
     Description=Prometheus System Monitoring and Alerting software. More @ https://prometheus.io
     [Service]
     User=root
     WorkingDirectory=/home/pi/prometheus
     ExecStart=/home/pi/prometheus/prometheus --config.file=prometheus.yml
     KillMode=mixed
     [Install]
     WantedBy=multi-user.target

5. Commit hand-built services to Dietpi memory so everything fires up @ boot and is controlled in one place

   • chmod service files ... I instinctively use 644 for these
     sudo chmod 644 /usr/lib/systemd/system/maltrail-server.service
     sudo chmod 644 /usr/lib/systemd/system/maltrail-sensor.service
     sudo chmod 644 /usr/lib/systemd/system/prometheus.service

   • run dietpi-services utility to add Maltrail server and sensor and Prometeous:
     // Use "Add Missing Service" option, pretty simple //

Re-Boot-A-Roo to check things out ....

Traffic Monitor data @ http://<ip>:8001
Prometheous should be @ http://<ip>:9090/metrics
Grafana (if installed) will be @ http://<ip>:3001 

--------- Odd's n ends from initial work ... I may still salvage something from here ---

Tested a bunch of stuff.   Attempting to make a little gizmoPi that watches over our quickly growing collection of IoT "things".

Redoo-a-Roo #2! Latest will be above this from now on!

---------

NTOPng - I forgot about this guy! It works OK, but pretty much becomes useless after a short period of time (10 mins) . It is nice, but not worth subscription price for our home needs.

Install - started @ https://packages.ntop.org/ and downloaded RPI via:

wget https://packages.ntop.org/RaspberryPI/apt-ntop_1.0.190416-469_all.deb

sudo dpkg -i apt-ntop_1.0.190416-469_all.deb

sudo apt-get install ntopng nprobe n2n

Moving on to installation guide @ https://www.ntop.org/guides/ntopng/what_is_ntopng.html#installing-on-linux

Install DOES permanently install ... ntopng will start when the system is rebooted an sits right on top of grafana (port 3000).

// I'll shut it down manually when working on maltrail/netmon stuff //

------

Maltrail - Sing-along-with https://computingforgeeks.com/setup-maltrail-malicious-traffic-detection-system-on-linux/

Maltrail Readme is VERY good too ... find it @ https://github.com/stamparm/maltrail/blob/master/README.md

Install Notes - rough for me? So I can retrace my steps

• DietPie bullseye for test ... installed stuff as root(?!)

  • Dietpi-config

    • Passwords/hostname ... yada yada

    • enabled both eth0 and wlan0 (wifi config country, etc)

• Used 32g sd - hopefully big enough

• had to install apt-utils (dietpi side effect)

• Also installed net-tools to get netstat and ifconfig

  • ifconfig'd wlan0 and eth0 to be promiscuous (ifconfig eth0 promisc)

  • netstat -i to check - see "P" in flg column

• Also installed python3 instead of 2 ... python3-pcappy, pip

• Install recap
  sudo apt-get install schedtool
  sudo apt-get install git python3-pcapy -y
  git clone https://github.com/stamparm/maltrail.git

• Couple of conf mods

  • HTTP_ADDRESS = IP to listen on for GUI (I used eth0 address)

  • Admin password - under USERS.
    To change: create a new one using echo -n '<Strong Password goes here>' | sha256sum | cut -d " " -f 1

  • Set DNS to google (per guide) - modified /etc/dhcpcd.conf

• Start via 

  • • 1.  server/sensor.py &

• to run test install dnsutils

• running from root/maltrail/maltrail/ ...

• ui @ http://<ip or FQDN>:8338

Test using mirrored IoT switch port & promisc wifi ... We'll see!?

// Seemed to run ok, but ... can't recall why I did a re-doo-a-roo but I did! //

----------- Part 2 - IoT traffic Monitor PLUS maltrail ---------

RaspOS lite as base, more network traffic tools then maltrail

1. download RaspOS lite and flash (I got bullsye)

2. Update - whoops "InRelease changed" warnings are annoying when 'official' release is used,
   sudo apt-get update --allow-releaseinfo-change
   sudo apt-get upgrade

3. Sing-a-long with https://www.technicallywizardry.com/raspberry-pi-network-monitor/ (read carefully)
   No huge struggles thru testing .... Help along the way:
   - https://danielmiessler.com/study/tcpdump/
   - Commandline flags @ https://github.com/zaneclaes/network-traffic-metrics#configuration

4. Test run - remember to check promisc (netstat -i) ... we'll need to make this stick before the dust settles

   • Explicitly set Port and Interface since I plan (hope) to run two of these - one on wifi, other on mirrored ethernet
     for WiFi:
     sudo python3 ./network-traffic-metrics.py "(src net 192.168.x.0/24 and not dst net 192.168.x.0/24) or (dst net 192.168.x.0/24 and not src net 192.168.x.0/24)" -i wlan0 -p 8000

   • Test via browser @ http://<IP>:8000/metrics

5. Install Prometheus - help @ https://prometheus.io/docs/prometheus/latest/getting_started/

   • Download w/ help from https://pimylifeup.com/raspberry-pi-prometheus/

     1. Check latest version (arm7) @ https://prometheus.io/download/

     2. Insert version number and Download via
        $ wget https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-armv7.tar.gz

     3. Unzip via (also rename directory in hopes of simplification)
        tar xvfz prometheus-*.tar.gz
        mv prometheus-2.x.x.linux-armv7/ prometheus/

   • Modify prometheus.yaml ... added external_lable to global and localhost:8000 target initially (run on same raspi)
     
     global:
     scrape_interval: 15s # How frequently to report
     # TPed was here: added external_labels
     external_labels:
     monitor: 'network-traffic-metrics'
     scrape_configs:
     - job_name: 'network-traffic-metrics'
     static_configs
     # TPed was here, change target to Network traffic Metrics on localhost for now
     # - targets: ["localhost:9090"]
     - targets: ["localhost:8000"]

   • Test by firing up prometheus:
     ./prometheus --config.file=prometheus.yml

Then surfing to metrics via:

http://<ip>:9090/metrics

6. Install Grafana - official docs @ https://grafana.com/docs/grafana/latest/installation/debian/. Even better, a tutorial @ https://grafana.com/tutorials/install-grafana-on-raspberry-pi/

   • From Wizardly page:
     sudo apt-get install -y apt-transport-https
     sudo apt-get install -y software-properties-common wget

   • wget in next step seemed to fail ... but I think it was just a warning: "apt-key Is Deprecated" message ... If it is more than a warning, this may help: https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-how-to-add.html:
     wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
     echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
     sudo apt-get update
     sudo apt-get

   • FINALLY Install grafana ... Some setup hints popped out during install:
     sudo apt-get install grafana
     ... yada ... yada ...
     Unpacking grafana (8.3.2) ...
     Setting up grafana (8.3.2) ...
     Adding system user `grafana' (UID 110) ...
     Adding new user `grafana' (UID 110) with group `grafana' ...
     Not creating home directory `/usr/share/grafana'.
     ### NOT starting on installation, please execute the following statements to configure grafana to start automatically
     using systemd
     sudo /bin/systemctl daemon-reload
     sudo /bin/systemctl enable grafana-server
     ### You can start grafana-server by executing
     sudo /bin/systemctl start grafana-server

   • Started grfana server - just like it told me to ...
     sudo /bin/systemctl start grafana-server.service

   • Surf to new grafana server via http://<IP or localhost>:3000, login as admin/admin and change PW

7. Now hook it all together - add prometheus as data source to grafana. All steps below because I haven't set anything up to automatically run yet...also need to remember where I put everything ....

netstat -i // Make sure interfaces are in promiscuous mode //

cd ./network-traffic-metrics

sudo python3 ./network-traffic-metrics.py "(src net 192.168.xx.0/24 and not dst net 192.168.xx.0/24) or

(dst net 192.168.xx.0/24 and not src net 192.168.xx.0/24)" -i wlan0 -p 8000 &

cd prometheus

./prometheus --config.file=prometheus.yml &

test python via http://<ip>:8000

test prometheus via http://<ip>:9090

8. ALMOST FINALLY - add Prometheus as datasource to grafana ... more @ https://grafana.com/docs/grafana/v7.5/datasources/prometheus/
   there was a panel for this *or* click gear on left and "datasource"
   Named the datasource Prometheus
   Changed HTTP to point to prometheus service @ http://<ip>:9090
   save & test

9. FINALLY! Added whats-his-name's dashboard (from original network-monitoring metrics) ... using +, Dashboard on left menu and add dashboard 12619

10. POOF! It works! I'll be damned!!

11. To Do's based on initial observations ...

    • For our world:

      1. To see ALL IoT THINGS, I'll have to mirror an ethernet port to catch wired THINGS.
         Will try to fire-up a second python metrics gatherer w/ different port to watch eth0 & add to prometheus as another target

         • Second python script seems fine - used a different port
           Started both with shell scripts - I want to see console output. Some [Skips] that I may want to filter out of tcpdump

         • Prometheus - second target: I just added second target to the prometheus.yaml file - because I didn't know any better!
           - targets: ["localhost:8000", "localhost:8080]

      2. TOO MANY IP's! Ill try a hosts file on the monitor/spypi to see if I can name the hosts in the display. Our IoT stuff is predictable
         Seems to be getting resolved at the python level, probably fine ... BUT, nothing shows in grafana when I send it resolved IP addresses
         soooo. in grafana dashboard, I replaced Regex in LocalIPs variable with: [A-Za-z0-9\.\-]{0,} (letters, numbers, dots & dashes)
         Previously it was: /^((127\.\d+\.)|(10\.\d+\.)|(172\.1[6-9]\.)|(172\.2[0-9]\.)|(172\.3[0-1]\.)|(192\.168\.)\d+\.\d+)$/
         (rfc1918 private addresses, I think)

      3. Decide if I will run all of this on a single RPI (including the maltrail thingy)

Bookmark collection

More info @ https://haxf4rall.com/2018/06/13/maltrail-malicious-traffic-detection-system/ (some popup email collector, tho)

InternetPi on top ... from https://github.com/geerlingguy/internet-pi

target tools @ https://sectools.org/ + https://sectools.org/tag/traffic-monitors/ (old)

Net Monitor w/ Prometheus @ https://www.technicallywizardry.com/raspberry-pi-network-monitor/

Static Domain servers @ https://www.thegeekpub.com/18336/change-the-raspberry-pi-dns-settings/

Delete stuff from Prometheus in https://www.shellhacks.com/prometheus-delete-time-series-metrics/

• Admin mode: run w/ --web.enable-admin-api

• curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}'

• kill -TERM <prometheus#>