Splunking on Pi – DIY Active Sensors [UPDATE: Now working with the WLANPi!]

Update: Now WLANPi compatible

I’d love to take the credit, but I can’t, @WifiNigel has been hard at work making his WiPerf tool work cross platform, and therefore on the WLANPi. Please scroll down to updated instructions, they header is nice and red so should stand out!
One important thing to note is the field headers on the latest commit have changed, which should be the last time – if you have installed this previously and configured Splunk Dashboards already, you may have to adjust them.

There are a few different flavour of sensor on the market; you have sensors which perform active tests, you have sensors which sniff frames, you have sensors which capture spectrum data, you even have sensors which sit on your wired network and try figure out how your network is performing.. blah, blah blah.

Don’t get me wrong, some are really good, but they all have a few things in common:
1) Fairly substantial up front cost
2) Subscription based
3) Proprietary
4) Require substantial change, whether it be appliances, cablers, downtime, etc.

So, what if I told you that there was a way to get an active sensor without a subscription? What if I told you you could get sensors that had an upfront cost of less than Β£50/$50? What if I told you that it wasn’t proprietary, and could be built on by the community to bring in new features? What if I told you the leg work had already been done, and the instructions are right here, in this blog?

Huh? Have I got your attention? Good, read on!

Here’s a bit of background.. shortly after I published my latest blog I got chatting to WiFiNigel. See, Nigel had a nifty bit of code that ran speed test outputs to a file, however he didn’t have an elegant way of logging that data to visualise – In my eyes, Splunk was the perfect tool to do that. So, Nigel did the easy bit (coding) and I did the hard bit (made pretty graphs). Very soon we saw the potential of Mr Bowdens labour, and we soon realised just how beneficial this could be.

Utilising a range of tools, we ultimately designed a sensor solution that ran on Raspberry Pi’s, and could connect over a VPN (ZeroTier) to log data to a central Splunk server (under my desk in my home office). We tested for a while in a production environment and I introduced some additional tools, such as Watchdog (to auto-reboot devices on network/other failure) and Apache Guacamole (which can be used to remotely and securely access devices outside of the ZeroTier VPN). Nigel’s python skills meant that we have quickly been able to implement several tests, such as:

  1. Speed test
  2. Ping Test of multiple hosts
  3. UDP iPerf Test
  4. TCP iPerf Test
  5. DNS resolution timer of multiple hosts
  6. DHCP timer
  7. Data Rate, RSSI, TX retransmits (for your Pi) and BSSID, SSID, etc

All of these tests can be turned on and off individually, and configured to test your favourite hosts, exportable in either JSON or CSV – and the code is on GitHub for you to pull/clone/branch/merge to your hearts content – WiPerf.
And I guess thats the key thing, the reason this has excited us as much as it has is because of the success of the WLANPi, the community have built a home grown tool that rivals any vendor tool, and the potential of a community built and driven distributed sensor network is huge.

So, what am I waiting for? Well.. here’s what you need

  1. RPI 3B+ or 4B (Older models are 2.4GHz only, you can still use with an external NIC)
  2. A device capable of running a Splunk Free server (https://docs.splunk.com/Documentation/Splunk/8.0.0/Installation/Systemrequirements)
  3. Thats it πŸ™‚

New: Cross Platform Instructions for WLANPi/RPI

  1. Install Splunk on a server/laptop/desktop. By default it will install Splunk Enterprise, you can convert to Splunk Free (500MB/day data cap) once installed.
    1. Link to Download: https://www.splunk.com/en_us/download/splunk-enterprise.html
    2. Link to Install Manual: https://docs.splunk.com/Documentation/Splunk/7.2.4/Installation/Whatsinthismanual
    3. After installation, go to Settings > Data Inputs
    4. Click HTTP Event Collector
    5. Click Global Settings and configure as per the below:
      All Tokens: Enabled
      Default Source Type: _JSON
      Default Index: Default
      Default Output Group: None
      Use Deployment Server: Unticked
      Enable SSL: Ticked
      HTTP Port Number: Up to you, but default is 8088 >> This is needed for your wiperf config file
    6. Click Save, then click New Token, with the following settings on the initial page:
      Name: Anything you want, haven’t worked out what this is used for
      All Other Fields: Blank/Default
    7. Click Next for the secondary page, and configure as follows:
      Source Type: Automatic
      App Context: Searching & Reporting
      Index: Add all
      Default Index: Main
    8. Click Review, then Submit. On the next screen it will display your HEC Token >> This is needed for your wiperf config file

      Your Splunk server is now configured, you can re-use your Token for all your sensors!
  2. Install Your Pi OS: Raspberry Pi
    1. Link to Download: https://www.raspberrypi.org/downloads/raspbian/
    2. Link to Install Manual: https://www.raspberrypi.org/documentation/installation/installing-images/README.md
    3. Important, before putting the SD Card into the Pi, create a file in the boot folder of the media called ssh (no extension). This will allow SSH access as soon as it is connected to a network.
  3. Install your Pi OS: WLANPi
    1. Visit http://wlanpi.com to get the image and write to an SD card.
  4. Insert the SD Card, plug into a network and power.
  5. Go to the WiPerf github repo and follow the README to install the scripts: https://github.com/wifinigel/wiperf
  6. Set the general section of the config.ini file, set your architecture to the correct platform, set data_format to _json, set data_transport to hec, and data_host to your Splunk server IP, and data_port and splunk_token to the relevant information from your Splunk server captured above.
  7. HEC will use your devices hostname as the default host in Splunk, so set this to something unique and meaningful otherwise all your data will be from host=raspberrypi

    Thats it, you should now be receiving data into Splunk. Scroll down past the UFW instructions for more Splunky goodness

Instructions for Slunk UFW
This is still a valid method, however more complex to install and does not work on the WLANPi

  1. Install Splunk on a server/laptop/desktop. By default it will install Splunk Enterprise, you can convert to Splunk Free (500MB/day data cap) once installed.
    1. Link to Download: https://www.splunk.com/en_us/download/splunk-enterprise.html
    2. Link to Install Manual: https://docs.splunk.com/Documentation/Splunk/7.2.4/Installation/Whatsinthismanual
    3. After installation, you need to add a Receiving Port: Click Settings > Forwarding & Receiving > under Configure Receiving click Add New, and enter a port number (I used the suggested port, 9997)
  2. Install Raspbian Lite on RPI
    1. Link to Download: https://www.raspberrypi.org/downloads/raspbian/
    2. Link to Install Manual: https://www.raspberrypi.org/documentation/installation/installing-images/README.md
    3. Important, before putting the SD Card into the Pi, create a file in the boot folder of the media called ssh (no extension). This will allow SSH access as soon as it is connected to a network.
  3. Insert the SD Card, plug into a network and power.
  4. Install the Splunk Universal Forwarder on the RPI (ARMv6) – this is the software used to forward data into Splunk: Link to Download: https://www.splunk.com/en_us/download/universal-forwarder.html (hint, once you start the download you can view a wget link on the right – you can copy this and download directly onto your RPI)
  5. This is the most problematic install, but this got it working for me – just run these commands:sudo tar xvzf splunkforwarder.....64.tgz -C /opt
  6. Run sudo ./splunk start from /opt/splunkforwarder/bin
  7. Hopefully, it should ask you to accept an EULA and set a username and password, if that works, you’re in!
  8. Go to the WiPerf github repo and follow the README to install the scripts: https://github.com/wifinigel/wiperf
  9. Set the WiPerf config.ini to write JSON files
  10. Head over to https://github.com/Krisalexroberts/wiperf-splunk-config and copy the .conf files into /opt/splunkforwarder/etc/system/local
  11. Set a unique sensor name on row 2 of inputs.conf and insert your Splunk server IP address in outputs.conf. If you don’t set a unique name for your sensor, all the data you receive will look like it came from the same host!
  12. In /opt/splunkforwarder/bin run sudo ./splunk enable boot-start
  13. Reboot!

That should be enough to get data into Splunk. Now the interesting bit is actually displaying the data – if you head over to Splunking on Pi you can see some example searches and how to display data, these searches can be inserted in Dashboard Panels.

The great thing about this is it barely uses any data, with Splunk free you get 500MB of data a day, with 5 sensors running tests every 2 minutes I’m using 0.1% of that allocation!

Anyway… thats how you get one sensor talking to Splunk. How about.. I don’t know.. 100?

ZeroTier

If you haven’t used ZeroTier before, you’re missing out.. so, what is it? Well, think of it as a big switch in the sky. It’s basically an overlay network which allows nodes to talk to eachother securely with minimal config, and for 100 nodes in a network, it’s free.

  1. Head over to https://zerotier.com/ and create an account
  2. Despite being free, you need to give your card details – but you’re not going to be charged, don’t worry!
  3. Click Networks, and Create a Network
  4. Once created, you will see it populate in the list, click on it – it will display a Network ID – thats the important bit.
  5. Click on Download and on your Sensors and your server, download the agent. Once downloaded and installed, on each node type sudo zerotier-cli join networkID or enter it in the relevant ZeroTier GUI on your server
  6. Back on the ZeroTier website, navigate back to your networks and scroll down, you’ll see a list of all the devices that want to join, tick the box and you’re done – everything’s talking. Remember to give devices a meaningful name!
  7. On your Splunk server, run ifconfig and you should see a new interface for your VPN, take a note of its IP address – you need to head back over to outputs.conf on your Sensors and replace the previous IP with the ZT IP.
  8. Once you’ve changed outputs.conf, navigate to /opt/splunkforwarder/bin and run ./splunk restart

Again.. thats it, that will allow up to 100 nodes to talk to each other – easy as pi! One problem I did run across was the encapsulation overhead on some networks makes it difficult to access the sensors via CLI. I had to set up a cronjob to set the ZeroTier MTU down to 1200 bytes (sudo ifconfig zerotierinterface mtu 1200) – thats obviously network independent though – just watch your toxic tails!

One thing to watch out for is that the DHCP reset in WiPerf doesn’t work well with ZeroTier as it has to re-establish a tunnel afterwards.

Watchdog

One thing I spotted pretty quickly was that the sensors went offline sometimes.. and quite randomly. I quickly worked out that RPIs aren’t great at handling network disconnections, so decided to install Watchdog to automatically reboot in preconfigured conditions.

  1. On your Pi run sudo apt-get install watchdog
  2. Edit the config file (sudo nano /etc/watchdog.conf)
  3. You can set watchdog to ping a host, and time out after x failures by setting the ping field to an IP/Host, interface to your wlan interface, and retry-timeout to whatever value you want – I found that anything less than 60 caused frequent reboots.

Apache Guacamole

Apache Guacamole is one thing that gets me excited, see the project here: https://guacamole.apache.org/

What is it? Well, it’s a web server that will let you SSH/RDP/etc to hosts from a web interface. Whilst it’s not really necessary for this project, I found that having multiple devices to manage was cumbersome. You’re best spinning up a CentOS server and joining it direct to ZeroTier, then running the install script found here: https://github.com/Zer0CoolX/guacamole-install-rhel

If you configure TOPT (2FA) and LetsEncrypt (SSL/TLS) via the script, you get really easy and straight forward secure access to a number of devices

As of today, this doesn’t work on the WLANPi – we simply can’t get Splunk installed, but there are plans to output the WiPerf data as Syslog which should allow you to output to Splunk (hence the unused transport config items in config.ini).

As the RPI runs a similar architecture to the WLANPi, you could in theory run most applications you can on a WLANPi on an RPI, eg Kismet – it’s pretty much a blank canvas though and we encourage you to share ideas and develop your own code and use cases!

Cisco Blog Awards
Vote for Nigel!

Personally, I think this is super cool, and the possibilities are endless and could quite easily rival a vendor tool – imagine kicking off a HORST capture when a threshold is breached, or mapping your sensors based on available BSSIDs using wigle.net – and they’re just the two I’ve thought of whilst writing this conclusion! I haven’t even begun to think about any integration between your WLC data and the data from the sensors!

Most of the effort here is from WiFiNigel – like I said, all I did was make pretty graphs. Nigel has been nominated in the Cisco 2019 IT Blog Awards, please repay him for not only his effort here, but his effort in all things wireless, by voting here, and of course read his blog here

Thanks!

One thought on “Splunking on Pi – DIY Active Sensors [UPDATE: Now working with the WLANPi!]

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.