A while back I debuted a new tool that I had been helping @WiFiNigel to develop called Wiperf. You can read all about WiPerf here on Nigel’s blog or on github. What I haven’t really discussed is the fact that I have been using Wiperf privately for over 6 months now, and it is incredibly useful and powerful, but dealing with remote sensors isn’t always as straight forward as you might think and sometimes it’s pretty hard keeping them online.
One of the massive benefits of ‘vendor’ sensor solutions is that they have a management platform, and in it’s simplest form, although a brilliant piece of work, Wiperf is a collection of scripts to test the network – there is no simple way of managing remote Pi’s that I have yet discovered.
As you all know, the wireless community is very active with some pretty bright and clever people, so some of todays problems probably won’t exist in the future; I will do my best to keep this up to date as the collective finds solutions.
Provisioning
First off, provisioning is a challenge. Whilst with a bit of attention and time you could probably code it, all the probes need configuring. First you have to get the SD card provisioned, then you have to get it on a network. If you’re local to a sensor, thats easy, but if you’re remote, you’re going to struggle.
My method at the moment of provisioning devices is using a private GitHub repo with the required config files – as an example:
sudo apt-get update && sudo apt-get upgrade -y
curl -s 'https://raw.githubusercontent.com/zerotier/ZeroTierOne/master/doc/contact%40zerotier.com.gpg' | gpg --import && \ if z=$(curl -s 'https://install.zerotier.com/' | gpg); then echo "$z" | sudo bash; fi
sudo zerotier-cli join <ztkey>
sudo apt-get install python3 python3-pip iperf3 git watchdog unattended-upgrades msmtp msmtp-mta bsd-mailx -y
sudo pip3 install iperf3 speedtest-cli configparser
sudo pip3 install git+git://github.com/georgestarcher/Splunk-Class-httpevent.git
cd ~
sudo git clone https://github.com/wifinigel/wiperf.git
sudo curl -s https://<gitkey>@raw.githubusercontent.com/<myprofile>/<supersecretrepo>/master/config.ini -o ~/wiperf/config.ini
sudo curl -s https://<gitkey>@raw.githubusercontent.com/<myprofile>/<supersecretrepo>/master/watchdog.conf -o /etc/watchdog.conf
sudo apt-get update && sudo apt-get upgrade -y
sudo systemctl enable watchdog
sudo curl -s https://<gitkey>@raw.githubusercontent.com/<myprofile>/<supersecretrepo>/master/mycron -o mycron
sudo crontab mycron
sudo curl -s https://<gitkey>@raw.githubusercontent.com/<myprofile>/<supersecretrepo>/master/50unattended-upgrades -o /etc/apt/apt.conf.d/50unattended-upgrades
sudo tee -a /etc/msmtprc > /dev/null <<EOT
account mailgun
host smtp.eu.mailgun.org
port 587
from $HOSTNAME@<supersecretdomain>
user postmaster@<supersecretdomain>
password <supersecretpassword>
auth on
tls on
tls_starttls on
tls_certcheck on
logfile ~/.msmtp.log
account default : mailgun
EOT
echo "This is the email body" > /tmp/body.txt && sudo mailx -s "This is the subject" me@<supersecretdomain> < /tmp/body.txt; rm /tmp/body.txt
sudo rebootThe above script isn’t really a script, it’s just a collection of commands – until I pick up a book and find some time, it’s how I provision. The benefit is that the only things I have to do is get a network connection locally and give a hostname – then copy and paste that into the Pi and we’re all good.
From top to bottom, this is what’s going on:
- Update and Upgrade via apt
- Install ZeroTier and join my Network
- Install the Wiperf pre-reqs, Watchdog, unattended-upgrades and an SMTP client (more on these later)
- Install Wiperf pre-reqs from Pip
- Clone the Wiperf repo
- Grab my Wiperf config
- Grab my Watchdog config
- Update and Upgrade via apt, again, just for good measure
- Enable Watchdog as a service
- Grab my crontab config
- Grab my unattended-upgrade config
- Configure SMTP using Mailgun
- Test Mailgun
- Reboot
All being well, your sensor is rocking and rolling by this point – it’s rough, nasty and crude but it works. Eventually I want to build an automated provision where you just feed a seed file to the Pi, but I’m not that clever.
Oh, and if you have a captive portal, you need to put additional thought into how you log in.
watchdog
Nigel has done a brilliant job of making Wiperf resilient, so Watchdog may not give you anything, but for me it’s ‘another line of defence’. Wiperf already reboots the sensor given certain conditions, and Watchdog pretty much does the same. Basically, it’s a daemon which runs and conducts predefined tests, depending on the outcome of those tests it can re-initiate a reboot. I use it quite simply, if a Pi cannot contact the Splunk server over ZeroTier it will reboot, but you can set many more tests, as defined here.
My config is pretty straight forward, I just amend the top section:
ping = zt.ip.add.here
#ping = 172.26.1.255
interface = ztinterface
#file = /var/log/messages
#change = 1407Unattended Upgrades
So, you have all your Pi’s around your network, across countries or whatever, how do you update them? Doing this manually would be a nightmare. Unattended-Upgrades just updates packages via apt automatically – again, it’s configurable – this guide roughly describes it.
For me, email updates were key. I opted to use mailgun’s free tier to email me with the changelog and notify of any updates. You can do this by editing the following section of the config file:
// Send email to this address for problems or packages upgrades
// If empty or unset then no email is sent, make sure that you
// have a working mail setup on your system. A package that provides
// 'mailx' must be installed. E.g. "user@example.com"
Unattended-Upgrade::Mail "email@goes.here";On top of that, you need the capability to send email from your Pi, thats the msmtp msmtp-mta bsd-mailx bit of my apt-get install above. On top, I configure the msmtp file, which is the last section of my ‘script’ above – finally, it sends a test email using the hostname@domain.
ZT MTU & CRONTAB
ZeroTier derives its MTU from the physical NIC, which you will often see is 2600 bytes. This works, most of the time, but as soon as you start tunnelling this via CAPWAP/EoIP or IPSec you may see performance problems. I haven’t found an elegant way of fixing the MTU permanently, so I do this via cron:
*/2 * * * * /usr/bin/python3 /home/user/wiperf/wi-perf.py >> /home/user/wiperf/wi-perf.log 2>&1
*/5 * * * * /sbin/ifconfig ztnfapqn4s mtu 1200
@reboot /sbin/ifconfig ztnfapqn4s mtu 1200Device types
So far, I’ve experimented with various models of Raspberry Pi – some of them never work as well as others. The most stable I have found is the RPI 4B+ 4GB model without an external NIC, and RPI 3B+ with a Comfast CF-912ac.
I have tried 3A+s, Zeros, etc, none of them seem as performant and stable. To put it one way, I have had 3x 3B+s online with the Comfast NICs permanently for over 6 months, they have never missed a beat – my 3A+ worked fine for a few days before it started struggling. I guess the point is you need to pick your Pi wisely and test it – when these are deployed remotely it can be difficult to rectify any errors. The below is a view of my 3A+, which is quite clearly struggling.
The other thing to consider is client capability, even the best RPI is a 1×1:1 device, so don’t expect anything a better MCS rate than MCS8 (86.6Mbps on a 20MHz channel with SGI)
Recovery Processes
If you rush into deploying these without any real recovery process you’re going to end up with some stranded Pi’s, and nobody likes a stranded Pi. We’ve heard people running into every problem you can imagine – building closures, dodgy config changes, things going missing, corrupt SD cards, each of which has left people wandering around remote buildings trying to figure out where they had actually put them and trying to get them online. Even the model of device is important, if you forget you’ve deployed an RPI 4B+ and rock up with a micro-USB cable, you’re not going to have a difficult time. It can be a pain. If you want to deploy Wiperf sensors in anger make sure you document their location and have a process for manual recovery!
Cyber Security
If you’re deploying these sensors, tell your Cyber Security team, otherwise they may end up being picked up and confused as a Russian spying device. One approach was to run purely over a guest wireless network using ZeroTier, so no data sent actually transits the network or internet natively and the underlaying wireless network almost becomes irrelevant – it’s just another client doing a network test and sending it over the internet, just like Ookla and Speedtest.net, who also collect stats from the tests. If you are working for an organisation which is security sensitive make sure they are involved and happy to sign off their deployment.
Conclusion
These sensors are very useful for an extremely low cost point, but I guess the whole point of this blog is that you need to have a think about how you want to design your solution. I would not recommend buying 10s of sensors without ensuring there is a management wrap around them, and without testing. Once you get it right, they can make a big difference to how you manage your network, both from day to day operations to assessing impact of change. Rushing into deploying sensors like this will cause you issues.
As mentioned in the intro, as the tools mature we will discover new tools and between us we will design more elegant solutions to provision and manage the devices – I have tried some linux MDM solutions but none of them really blew me away or offered anything, but it is still early days and there is a lot of water to flow under the bridge before I would class Wiperf as an ‘enterprise ready’ solution.
Beacon Sandwich 
You must be logged in to post a comment.