Raspberry PI IoT Network Monitor
Work in Process ... see status & log
The goal of the SentinelPi is to watch-over our world of IoT smart devices and to learn about traffic patterns. Keep an eye on who is talking to who and attempt to monitor behavior! Recipe will likely evolve as Network monitor tools are installed/tested and tuned.
This appears to be working - VERY surprised!
Fun one!
Better ideas are always welcome!
SentinelPi: A Pi to watch over home network of smart devices
Original: December 2021
Last Update: 28 April 2022 *** Still tweaking ***
- Prelim tests look amazingly good, built initial SentinelPi
- Network Traffic Monitor + Maltrail initially
Project Status
As always, Pi recipes are basically notes-to-self, used to retrace steps for upgrades, rebuilds and such - but may also be useful for others.
Initial tools - Maltrail & Network Traffic Monitor - seems to work well together!
Luckily, our smart 'Things' were contained to a separate subnet behind a somewhat smart switch that provides SentinelPi a peek into IoT traffic via mirror port.
Still some network config polishing to do. Still more tweaks to come ...
Level of Difficulty & Prep Time
This one is a tad complicated!
AT LEAST a day's worth of work plus some time to get things up, rolling and fine-tuned for your network,
Networking know-how (TCP/IP subnetting) and Linux knowledge is required.
Hardware
The SentinelPi environment is a combination of software, RPi hardware *and* cooperating network parts/features (built using old stuff laying around, of course!) ...
A semi-smart switch that supports port mirroring:
Like a TP-Link TL-SG108E*WiFi AP ... Old WiFi Router w/ AP mode (WRN2000).
Raspberry Pi 3 (seems fine with workload)
32g MicroSD (for starters)
Network hardware looks a bit like this:
Software
So far ...
<Possibly more to come here>
Software looks like this (I'll get better shot):
Step By Step
RaspOs Install/Config
Initially running on RPi 3 that was sitting here doing nothing - will watch carefully. Hopefully 32g SD will be enough, will watch this too!
Using RaspOS Lite on this one, mostly due to python scripts that appear to have been developed in this environment.
Guide is @ https://www.raspberrypi.com/documentation/computers/os.html
Download and Flash RaspOS Lite (Bullseye!). Raspberry Pi Imager is now easiest.
Initial Boot w/ Keyboard & Monitor attached
sudo raspi-config
System Options (kinda nice - main config options here now):
Wireless Lan: Set Country Code, select SSID/pw of local IoT Land
Password: Chang pi's pw
Set Hostname
Boot/Autologin: Console Autologin
Wait for Network @ Boot
Interface Options
Enable SSH
Localisation Options
Locale: en_US.utf-8 UTF-8
Timezone
Keyboard: US
WLAN Country Code (prob already set)
Advanced
Expand Filesystem (probably already done)
Update raspi-config tool
<reboot-a-roo!> and switch to SSH for the rest of this
Update/Upgrade ... yada yada
sudo apt update
sudo apt full-upgrade
Editor: Don't forget joe (or your favorite editor)
sudo apt-get install joe
Set Static IP addresses (v4) and google dns per Network-Traffic-Monitor guy's recommendation
interface eth0
static ip_address=192.168.xx.x/24
static routers=192.168.xx.x
static domain_name_servers=8.8.8.8 8.8.4.4
interface wlan0
static ip_address=192.168.xx.x/24
static routers=192.168.xx.x
static domain_name_servers=8.8.8.8 8.8.4.4
sudo crontabPut eth0 into promiscuous mode via crontab @reboot ...
// not sure this is THE correct way, but we'll see //
sudo crontab -e
// Append to file //
# TPed was here, put interfaces into promisc mode
@reboot sudo ifconfig eth0 promiscFinally turn off wireless stuff ... There's MORE than enough RF in our little network closet already!!
// reminder: rfkill survives through a boot ... to undo this use "unblock" //
sudo rfkill block wifi
sudo rfkill block bluetooth
Maltrail Install/Config
Sing-a-long with https://github.com/stamparm/maltrail/blob/master/README.md#quick-start. The Whole README is very good!
Install
sudo apt-get install git python3 python3-dev python3-pip python-is-python3 libpcap-dev build-essential procps schedtool
sudo apt-get install git python3-pcapy -y
sudo apt-get install dnsutils
sudo pip3 install pcapy-ng
git clone --depth 1 https://github.com/stamparm/maltrail.gitInitial Testing ...
Check Promiscuous mode (you should see "P" in flags)
netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 151201 0 0 0 70372 0 0 0 BMPRU
lo 65536 14 0 0 0 14 0 0 0 LRU
Fire up sensor ... it will take a bit as it updates trails, feeds or whatevers
Install and fire up maltrail server (gui)
git clone --depth 1 https://github.com/stamparm/maltrail.git
// I don't think this was needed //
sudo python3 server.py &Test via browser @ http://<ipaddress>:8338
Initial login is admin/changeme!
A couple of queries to get something logged (seem to be dangerous places! Test was in README)
ping -c 1 136.161.101.53
nslookup morphed.ruRefresh maltrail web interface to see logged threats
Complete Install and Auto-Start once we know this thing works!
Minimal config file touch-ups - /home/pi/maltrail/mailtrail.conf
Change password:
Create a new password like this:
echo -n '<NewStrongPassword>' | sha256sum | cut -d " " -f 1 1ee0cd0713ba2b4fde13881618e0995ee2f560c41fb7b2d195847e5a018fc03c2Edit maltrail.conf and plant your new password (:0: at end) in the USERS section, replace the default password (changeme!)
# User entries (username:sha256(password):UID:filter_netmask(s))
# Note(s): sha256(password) can be generated on Linux with: echo -n 'password' | sha256sum | cut -d " " -1
# UID >= 1000 have only rights to display results (Note: this moment only functionality implemented at the c
# filter_netmask(s) is/are used to filter results
USERS
admin:1ee0cd0713ba2b4fde13881618e0995ee2f560c41fb7b2d195847e5a018fc03c2:0: # NewStrongPassword
# local:9ab3cd9d67bf49d01f6a2e33d0bd9bc804ddbe6ce1ff5d219c42624851db5dbc:1000:192.168.0.0/16 # changeme!Bounce the server to make sure you did that right! Log back into GUI after bounce
cd /home/tech/maltrail
pkill -f server.py
python server.py &
REMEMBER! This is currently running in a terminal session! Not as a service, DON'T EXIT SSH OR CLOSE TERMINAL SESSION!
// See "Pull it all Together" section below - We will turn this into a Service that starts when the system boots //
Network Traffic Monitor Install/Config
Sing-a-long with https://www.technicallywizardry.com/raspberry-pi-network-monitor/ (read carefully)
Command line syntax & more in https://github.com/zaneclaes/network-traffic-metrics#readme
Installs (some of this was already installed with Maltrail, but we'll keep components independent)
cd /home/pi
sudo apt-get install git python3-pip tcpdump
sudo pip3 install argparse prometheus_client
git clone https://github.com/zaneclaes/network-traffic-metrics.git
cd ./network-traffic-metrics
A Script to fire up eth0 monitor. Season tcpdump filters to taste (192.168.x.0/24) and append with a "&" to run in background ... May decide to run these as services
sudo python3 -i eth0 -p 8001 ./network-traffic-metrics.py "(src net 192.168.x.0/24 and not dst net 192.168.x.0/24) or (dst net 192.168.x.0/24 and not src net 192.168.x.0/24)"Test via browser @ given ports. You should see statistics
http://<ip>:8001
Prometheus Install/Config
Install Prometheus with help of https://prometheus.io/docs/prometheus/latest/getting_started/
cd /home/pi (don't install this under maltrail or network-monitor!)
Download w/ help from https://pimylifeup.com/raspberry-pi-prometheus/
Check latest version (arm7) @ https://prometheus.io/download/
Insert version number and Download via
$ wget https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-armv7.tar.gzUnzip (also rename directory in hopes of simplification)
tar xvfz prometheus-*.tar.gz
mv prometheus-2.x.x.linux-armv7/ prometheus/
Modify ./network-traffic-metrics/prometheus/prometheus.yml ... add localhost targets. Now looks like this ...
global:
scrape_interval: 15s # How frequently to report
external_labels:
monitor: 'network-traffic-metrics'
scrape_configs:
- job_name: 'network-traffic-metrics'
# Tped was here. target changed to localhost and target of 8001
static_configs:
- targets: ['localhost:8001'] # The Network Traffic Metrics IP/portTest it out
./prometheus --config.file=prometheus.yml
Then surf to metrics via:
http://<ip>:9090/metrics
Grafana Install/Config
Install Grafana - official docs @ https://grafana.com/docs/grafana/latest/installation/debian/.
Even better, a tutorial @ https://grafana.com/tutorials/install-grafana-on-raspberry-pi/
From Wizardly page:
sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wgetwget in next step seemed to fail ... but I think it was just a warning: "apt-key Is Deprecated" message ... If it is more than a warning, this may help: https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-how-to-add.html:
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get updateFINALLY Install grafana ... Some setup hints popped out during install:
sudo apt-get install grafana
... yada ... yada ...
Unpacking grafana (8.3.2) ...
Setting up grafana (8.3.2) ...
Adding system user `grafana' (UID 110) ...
Adding new user `grafana' (UID 110) with group `grafana' ...
Not creating home directory `/usr/share/grafana'.
### NOT starting on installation, please execute the following statements to configure grafana to start automatically
using systemd
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server
### You can start grafana-server by executing
sudo /bin/systemctl start grafana-server
Started grfana server - just like it told me to ...
sudo /bin/systemctl start grafana-server.serviceSurf to http://<ip>:3000 - login as admin/admin, wit will force PW change
Configure: Add datasource and Network Traffic Monitor guys's dashboard.
Click on gear in Grafana UI:Add Datasource->Choose Prometheus and set server to http://<ip of server>:9090
Add Dashboard: Clink on "+" then import Network Traffic Dashboard by number (12619)
set datasource to prometheus
Pull it all together and start everything @ boot
I'll want this contraption to come up by itself when the system boots.
(this may change ... seems clumsy, but it works. Will probably incorporate final AutoStart into above setup/config)
Here is the current order of events ...
Put eth0 into promiscuous mode via crontab @reboot ... // This seems to work //
sudo crontab -e
// Append to file //
# TPed was here, put interfaces into promisc mode
@reboot sudo ifconfig eth0 promiscNetwork Traffic Monitor Auto Start: Stick this in crontab @ reboot also ... per recommendation in doc:
(make sure python script is fully qualified!)
# TPed was here again! Fire up network traffic monitor @ boot
@reboot sudo python3 /home/pi/network-traffic-metrics/network-traffic-metrics.py -i eth0 -p 8001 "(src net 192.168.xx.0/24 and not dst net 192.168.xx.0/24) or (dst net 192.168.xx.0/24 and not src net 192.168.xx.0/24)" &Maltrail Auto Start after boot // This seems solid //
sensor.py: This will run as a service. "maltrail.service" provided w/ install and I can take a hint!
First copy the provided maltrail .service file to where all the other .service files live! (I checked grafana install)
sudo cp /home/pi/maltrail/maltrail.service /usr/lib/systemd/system/maltrail.serviceNext, Edit the .service file to set working directory to where I installed this thing. Now looks like this:
[Unit]
Description=Malicious traffic detection system https://github.com/stamparm/maltrail
[Service]
User=root
WorkingDirectory=/home/pi/maltrail
ExecStart=/usr/bin/python3 sensor.py
KillMode=mixed
[Install]
WantedBy=multi-user.targetpretty sure we need to chmod that dang thang ....
sudo chmod 644 /usr/lib/systemd/system/maltrail.serviceCheck our work ....
Kill whatever we have running ....
sudo pkill -f server.py
sudo pkill -f sensor.pyNow clear the deck and start the the sensor as a service
sudo systemctl daemon-reload
sudo systemctl enable maltrail.service
sudo systemctl start maltrail.service
server.py: Life seems to be good if we fire this up using rc.local so it starts after services. // seems to work fine //
Edit /etc/rc.local and append ... server using absolute path to be safe. Bottom of file now looks like this ...
# By default this script does nothing.
# Print the IP address
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
printf "My IP address is %s\n" "$_IP"
fi
# TPed was here. Fire up maltrail server (I'd think sensor.service is already running)
python3 /home/pi/maltrail/server.py &
exit 0
Grafana: Was originally installed as an Auto Start service
Prometheus Auto Start via https://sbcode.net/prometheus/prometheus-service/
This guy will be fired up from prometheus path so remember to have the correct prometheus.yml in the directory (/home/pi/prometheus/ in my case). Original yml was created under network-monitor thingy
Create service file @ /user/lib/systemd/system/prometheous.service
[Unit]
Description=Prometheus System Monitoring and Alerting software. More @ https://prometheus.io
[Service]
User=root
WorkingDirectory=/home/pi/prometheus
ExecStart=/home/pi/prometheus/prometheus --config.file=prometheus.yml
KillMode=mixed
[Install]
WantedBy=multi-user.targetThis should commit it all to memory for next boot *and* fire it up now
sudo systemctl daemon-reload
sudo systemctl enable prometheus.service
sudo systemctl start prometheus.service
Notes
Log
April 2022: Not bothering w/hostname resolution in Network Traffic Monitor, I'm used to the IP addresses now ... It's like watching digital rain!
Prometheus has been going into the ditch periodically - running our of memory? or something?
looks like this...
SentinelPi prometheus[392]: ts=2022-04-20T03:43:07.200Z caller=head.go:803 level=info component=tsdb msg="Head GC completed" duration=19.600777ms
SentinelPi prometheus[392]: ts=2022-04-20T03:43:07.367Z caller=db.go:830 level=error component=tsdb msg="compaction failed" err="compact head: head memory truncate: truncate chunks.HeadReadWriter: mmap, size 134217728: cannot allocate memory"
SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.264Z caller=compact.go:518 level=info component=tsdb msg="write block" mint=1650420006283 maxt=1650427200000 ulid=01G12NFH6X53RW9510QTTDGS9S duration=666.434323ms
SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.295Z caller=head.go:803 level=info component=tsdb msg="Head GC completed" duration=22.080236ms
SentinelPi prometheus[392]: ts=2022-04-20T05:00:22.298Z caller=db.go:830 level=error component=
tsdb msg="compaction failed" err="compact head: head memory truncate: truncate chunks.HeadReadWriter: write dat
a/chunks_head/000545: file already closed"
....
yada yada
....
prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
SentinelPi systemd[1]: prometheus.service: Failed with result 'exit-code'.
Prometheus version:
./prometheus --version
prometheus, version 2.31.1 (branch: HEAD, revision: 411021ada9ab41095923b8d2df9365b632fd40c3)
build user: root@b013bc8edd0b
build date: 20211105-20:27:14
go version: go1.17.3
platform: linux/arm
I see some chatter @ https://github.com/prometheus/prometheus/issues/7378
Updating everything
apt update
apt upgrade
Looks like there is also a prometheus update - 2.35 - via latest version (arm7) @ https://prometheus.io/download/. I scanned release notes from 2.32 to 2.35 for memory leaks and such ... maaaaybeeee?! I'll watch and update prometheous next month
ToDo: put together a SentinelPi Update procedure or script
---------------------
March 2022: Little setback. The AP (from junk drawer in my workroom) seems to have given up the ghost - like a VERY HARD RESET whilst it was running!!! Unrelated to SentinelPi software & such ... BUT VERY ODD!! NetGear WRN2000 factory reset appears to have changed its Serial Number & Default password - they are now different from those printed on the router @ factory! No $h1t?!?!! I can get it working, but it's running @ about 1/2 speed w/ no apparent symptoms (WiFi or Ethernet errors, retransmits, et
February 2022: A few more ToDo's
Polished up this recipe, the whole contraption now starts when SentinelPi is booted
Things came up MUCH better this time around - I booted most of the closet when I switched over to AP for WiFi things
// I'm guessing there were some ARPs cached or something that caused some angst the first time //ToDo: If/When I rebuild, it may be best to use DietPi for this one - better control over hardware and runtime world
Netmonitor: Still trying to figure out how to display hostnames rather than local IPs ... playing with -fqdn option - Not going to bother! I'm used to IP's now
January 2022. Project looks do-able ... a few hurdles and ToDo's and Fixed Stuff
Hostnames (as opposed to ip addresses) would be nice ... in both maltrail and traffic monitor.
Something is wrong (static route?) when running things via the AP ... e.g. HomePi SSH is broke! Maybe more! OK Now
Could I run even more monitors & tools? Wireshark-like thing? IDS, like snort? What else?
Make this thing restart-able. It's a B1tch to restart if I accidentally close my ssh session! Updating recipe as I go ...
Prometheus & Gafana were simple - just run them as systemd services
Maltrail May still need some work ... there was a systemd service template for the sensor but ... server needs to fireup at some point, I presume AFTER the sensor (which may be slow?). Firing up server.py in rc.local seems to be OK
Network-traffic-monitor agent/sensor thingies - created a couple of scripts for now, will need to be set up as services or cron @boot
Network configuration issues to deal with .... Promiscuous mode only works with THINGS that are wired ...
All Wired THINGS seems fine. SentinelPi sits on a port that receives mirrored Ethernet traffic from the port that heads to the ISP Router (Semi-Smart Switch)
WiFi THINGS: running second copy of agent /w different port to add in wifi traffic SEEMS to work ... BUT ... Promiscuous Mode only picks up broadcast traffic on our THINGS SSID ... Couple of things to try here
Need better understanding of WiFi Monitor Mode - what can a Pi do? Do I need external adapters, etc
Good Article @ https://networkengineering.stackexchange.com/questions/3774/using-wireless-cards-in-promiscuous-mode/3787#3787
No Guts/No Glory - add USB WiFi Dongle (I have a drawer-full of EDIMAX ). It looks like it's good to go for Monitor Mode BUT I still don't know if that would work
SentinelPi MAY need to be the main AP for these IoT world ... I think this would be easy (our IoT is on isolated SSID), but may take a bit more RPI horsepower
May be able to do RPi Monitor Mode tests on SuitiePi (RaspAP)
Investigate other probe-like configurations ... e.g. Netbook
Separate/Commercial WiFi AP is probably the best answer here. Proximity is a problem in our home, IoT devices seems to be spreading throughout the house. We already have areas with weak WiFi signals ... probably time for Mesh anyhoo
December 2021. Completed initial search for tools for this project.
Settled on maltrail + a couple of copies of Network Traffic Monitor python scripts running on a PI 3 I had laying around. PI has eth0 *and* wlan0 in promiscuous mode and is connected to our 'things' subnet via wifi and ethernet. The ethernet port is switch mirror of traffic destined for our ISP router and should pick up all traffic from wired 'things' on our network.
Initial software install/setup was all in foreground - no services, cron jobs or anything. I just left it all running in an ssh terminal session to see if it would all work together ... it seems to be! I've been watching cpu, disk space + whatever else I find to monitor. It seems to be working, I see traffic in the tools! I'm actually surprised!
Initial Fumblings
--------- Odd's n ends from initial work ... I may still salvage something from here ---
Tested a bunch of stuff. Attempting to make a little gizmoPi that watches over our quickly growing collection of IoT "things".
Redoo-a-Roo #2! Latest will be above this from now on!
---------
NTOPng - I forgot about this guy! It works OK, but pretty much becomes useless after a short period of time (10 mins) . It is nice, but not worth subscription price for our home needs.
Install - started @ https://packages.ntop.org/ and downloaded RPI via:
wget https://packages.ntop.org/RaspberryPI/apt-ntop_1.0.190416-469_all.deb
sudo dpkg -i apt-ntop_1.0.190416-469_all.deb
sudo apt-get install ntopng nprobe n2n
Moving on to installation guide @ https://www.ntop.org/guides/ntopng/what_is_ntopng.html#installing-on-linux
Install DOES permanently install ... ntopng will start when the system is rebooted an sits right on top of grafana (port 3000).
// I'll shut it down manually when working on maltrail/netmon stuff //
------
Maltrail - Sing-along-with https://computingforgeeks.com/setup-maltrail-malicious-traffic-detection-system-on-linux/
Maltrail Readme is VERY good too ... find it @ https://github.com/stamparm/maltrail/blob/master/README.md
Install Notes - rough for me? So I can retrace my steps
DietPie bullseye for test ... installed stuff as root(?!)Dietpi-configPasswords/hostname ... yada yadaenabled both eth0 and wlan0 (wifi config country, etc)
Used 32g sd - hopefully big enough
had to install apt-utils (dietpi side effect)
Also installed net-tools to get netstat and ifconfig
ifconfig'd wlan0 and eth0 to be promiscuous (ifconfig eth0 promisc)
netstat -i to check - see "P" in flg column
Also installed python3 instead of 2 ... python3-pcappy, pip
Install recap
sudo apt-get install schedtool
sudo apt-get install git python3-pcapy -y
git clone https://github.com/stamparm/maltrail.gitCouple of conf mods
HTTP_ADDRESS = IP to listen on for GUI (I used eth0 address)
Admin password - under USERS.
To change: create a new one using echo -n '<Strong Password goes here>' | sha256sum | cut -d " " -f 1Set DNS to google (per guide) - modified /etc/dhcpcd.conf
Start via
server/sensor.py &
to run test install dnsutils
running from root/maltrail/maltrail/ ...
ui @ http://<ip or FQDN>:8338
Test using mirrored IoT switch port & promisc wifi ... We'll see!?
// Seemed to run ok, but ... can't recall why I did a re-doo-a-roo but I did! //
----------- Part 2 - IoT traffic Monitor PLUS maltrail ---------
RaspOS lite as base, more network traffic tools then maltrail
download RaspOS lite and flash (I got bullsye)
Update - whoops "InRelease changed" warnings are annoying when 'official' release is used,
sudo apt-get update --allow-releaseinfo-change
sudo apt-get upgradeSing-a-long with https://www.technicallywizardry.com/raspberry-pi-network-monitor/ (read carefully)
No huge struggles thru testing .... Help along the way:
- https://danielmiessler.com/study/tcpdump/
- Commandline flags @ https://github.com/zaneclaes/network-traffic-metrics#configurationTest run - remember to check promisc (netstat -i) ... we'll need to make this stick before the dust settles
Explicitly set Port and Interface since I plan (hope) to run two of these - one on wifi, other on mirrored ethernet
for WiFi:
sudo python3 ./network-traffic-metrics.py "(src net 192.168.x.0/24 and not dst net 192.168.x.0/24) or (dst net 192.168.x.0/24 and not src net 192.168.x.0/24)" -i wlan0 -p 8000Test via browser @ http://<IP>:8000/metrics
Install Prometheus - help @ https://prometheus.io/docs/prometheus/latest/getting_started/
Download w/ help from https://pimylifeup.com/raspberry-pi-prometheus/
Check latest version (arm7) @ https://prometheus.io/download/
Insert version number and Download via
$ wget https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-armv7.tar.gzUnzip via (also rename directory in hopes of simplification)
tar xvfz prometheus-*.tar.gz
mv prometheus-2.x.x.linux-armv7/ prometheus/
Modify prometheus.yaml ... added external_lable to global and localhost:8000 target initially (run on same raspi)
global:
scrape_interval: 15s # How frequently to report
# TPed was here: added external_labels
external_labels:
monitor: 'network-traffic-metrics'
scrape_configs:
- job_name: 'network-traffic-metrics'
static_configs
# TPed was here, change target to Network traffic Metrics on localhost for now
# - targets: ["localhost:9090"]
- targets: ["localhost:8000"]Test by firing up prometheus:
./prometheus --config.file=prometheus.yml
Then surfing to metrics via:
http://<ip>:9090/metrics
Install Grafana - official docs @ https://grafana.com/docs/grafana/latest/installation/debian/. Even better, a tutorial @ https://grafana.com/tutorials/install-grafana-on-raspberry-pi/
From Wizardly page:
sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wgetwget in next step seemed to fail ... but I think it was just a warning: "apt-key Is Deprecated" message ... If it is more than a warning, this may help: https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-how-to-add.html:
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-getFINALLY Install grafana ... Some setup hints popped out during install:
sudo apt-get install grafana
... yada ... yada ...
Unpacking grafana (8.3.2) ...
Setting up grafana (8.3.2) ...
Adding system user `grafana' (UID 110) ...
Adding new user `grafana' (UID 110) with group `grafana' ...
Not creating home directory `/usr/share/grafana'.
### NOT starting on installation, please execute the following statements to configure grafana to start automatically
using systemd
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server
### You can start grafana-server by executing
sudo /bin/systemctl start grafana-serverStarted grfana server - just like it told me to ...
sudo /bin/systemctl start grafana-server.serviceSurf to new grafana server via http://<IP or localhost>:3000, login as admin/admin and change PW
Now hook it all together - add prometheus as data source to grafana. All steps below because I haven't set anything up to automatically run yet...also need to remember where I put everything ....
netstat -i // Make sure interfaces are in promiscuous mode //
cd ./network-traffic-metrics
sudo python3 ./network-traffic-metrics.py "(src net 192.168.xx.0/24 and not dst net 192.168.xx.0/24) or
(dst net 192.168.xx.0/24 and not src net 192.168.xx.0/24)" -i wlan0 -p 8000 &
cd prometheus
./prometheus --config.file=prometheus.yml &
test python via http://<ip>:8000
test prometheus via http://<ip>:9090
ALMOST FINALLY - add Prometheus as datasource to grafana ... more @ https://grafana.com/docs/grafana/v7.5/datasources/prometheus/
there was a panel for this *or* click gear on left and "datasource"
Named the datasource Prometheus
Changed HTTP to point to prometheus service @ http://<ip>:9090
save & testFINALLY! Added whats-his-name's dashboard (from original network-monitoring metrics) ... using +, Dashboard on left menu and add dashboard 12619
POOF! It works! I'll be damned!!
To Do's based on initial observations ...
For our world:
To see ALL IoT THINGS, I'll have to mirror an ethernet port to catch wired THINGS.
Will try to fire-up a second python metrics gatherer w/ different port to watch eth0 & add to prometheus as another targetSecond python script seems fine - used a different port
Started both with shell scripts - I want to see console output. Some [Skips] that I may want to filter out of tcpdumpPrometheus - second target: I just added second target to the prometheus.yaml file - because I didn't know any better!
- targets: ["localhost:8000", "localhost:8080]
TOO MANY IP's! Ill try a hosts file on the monitor/spypi to see if I can name the hosts in the display. Our IoT stuff is predictable
Seems to be getting resolved at the python level, probably fine ... BUT, nothing shows in grafana when I send it resolved IP addresses
soooo. in grafana dashboard, I replaced Regex in LocalIPs variable with: [A-Za-z0-9\.\-]{0,} (letters, numbers, dots & dashes)
Previously it was: /^((127\.\d+\.)|(10\.\d+\.)|(172\.1[6-9]\.)|(172\.2[0-9]\.)|(172\.3[0-1]\.)|(192\.168\.)\d+\.\d+)$/
(rfc1918 private addresses, I think)Decide if I will run all of this on a single RPI (including the maltrail thingy)
Bookmark collection
More info @ https://haxf4rall.com/2018/06/13/maltrail-malicious-traffic-detection-system/ (some popup email collector, tho)
InternetPi on top ... from https://github.com/geerlingguy/internet-pi
target tools @ https://sectools.org/ + https://sectools.org/tag/traffic-monitors/ (old)
Net Monitor w/ Prometheus @ https://www.technicallywizardry.com/raspberry-pi-network-monitor/
Static Domain servers @ https://www.thegeekpub.com/18336/change-the-raspberry-pi-dns-settings/
Delete stuff from Prometheus in https://www.shellhacks.com/prometheus-delete-time-series-metrics/
Admin mode: run w/ --web.enable-admin-api
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}'
kill -TERM <prometheus#>