WLAN Monitoring – Splunking on Pi

Splunking Around

The Caveat/Disclaimer

I’m going to start with some caveats that completely undermine myself and this blog, but I feel it is needed and I need to be completely honest:

  1. I am by no means an expert in Splunk
  2. I am not an expert in Wi-Fi
  3. I am not an expert in any of the tools we talk about here, including Linux

Why did I feel that was necessary? Well, some of the things I talk about here are simply my own way of discovering how to use these tools and how to get these tools to interface with each other, and they probably do not follow any industry standards or best practices.
This for me is a learning journey, and some of the documentation is lacking or out of date so I have found my own ways to overcome these challenges. To make things easier for myself, where the documentation is up to scratch I will simply refer to that, but I have had to deviate from some standard instructions and ‘bodge’ things.

Finally, this is evolving, I’ve only just got it working!

Now for the blog…

Ever since I started playing with Splunk at work I have become quite obsessed with it, as have most of my team. Seeing data displayed in different ways helps provoke new thoughts, ideas, and also helps identify issues that you never knew – it is an awesome bit of software and I encourage all WLAN professionals to have a go. On top of that, there are some great innuendos you can use playing with Splunk, which helps keep the work day light hearted.

Splunk

I’ve also been playing a bit with a WLANPi, and some of the tools like Horst and Kismet, and have been really impressed with the wealth of data that you can export – the WLANPi is one of the best collaboration efforts of the industry

Over a year ago I had a vision of both these worlds colliding and using a remote Pi to stream data back to Splunk, but I had numerous problems with the hardware and software that was available. For example, I could never get the Splunk Forwarding software installed on a WLANPi, and some problems with the RPI platforms that I couldn’t get around (until now)

When the RPI 4 came out, I decided to give it another go – here’s how you can.

What You Need

  1. RPI
  2. Server/Laptop/Desktop
  3. NICs capable of Monitor mode (I use Comfast CF-912AC)
  4. SD Card

Getting Installed

  1. Install Splunk on a server/laptop/desktop. By default it will install Splunk Enterprise, you can convert to Splunk Free (500MB/day data cap) once installed.
    1. Link to Download: https://www.splunk.com/en_us/download/splunk-enterprise.html
    2. Link to Install Manual: https://docs.splunk.com/Documentation/Splunk/7.2.4/Installation/Whatsinthismanual
  2. Install Raspbian Lite on RPI
    1. Link to Download: https://www.raspberrypi.org/downloads/raspbian/
    2. Link to Install Manual: https://www.raspberrypi.org/documentation/installation/installing-images/README.md
    3. Important, before putting the SD Card into the Pi, create a file in the boot folder of the media called ssh (no extension). This will allow SSH access as soon as it is connected to a network.
  3. Insert the SD Card, plug into a network and power.
  4. SSH onto the RPI (you may need to find its IP from a network scanning tool) and install the Re4son Kernel. This Kernel has the support for a heck of a lot of adapters and makes things easier:
    1. Download and Install Manual: https://re4son-kernel.com/re4son-pi-kernel/
  5. Install the Splunk Universal Forwarder on the RPI (ARMv6) – this is the software used to forward data into Splunk:
    1.  Link to Download: https://www.splunk.com/en_us/download/universal-forwarder.html (hint, once you  start the download you can view a wget link on the right – you can copy this and download directly onto your RPI)
    2. This is the most problematic install, but this got it working for me – just run these commands:
      1. sudo tar xvzf splunkforwarder.....64.tgz -C /opt
      2. Set the environment variable as documented here: http://dev.splunk.com/view/quickstart/SP-CAAAFDH
      3. ln -s /lib/arm-linux-gnueabihf/ld-linux.so.3 /lib/ld-linux.so.3
      4. Run sudo ./splunk start from $SPLUNK_HOME/bin
      5. Hopefully, it should ask you to accept an EULA and set a username and password, if that works, you’re in!
  6. Disable Predictable Network Names – this gets around an annoyance where the first adapter Raspbian sees becomes “wlan0”, so without disabling this, it can sometimes assign the wrong interface as wlan0 with an external NIC:
    1. Run sudo raspi-config
    2. Go to Option 2 – Network Options
    3. Go to Option 3 – Network interface Names
    4. Change to disable
    5. Exit the wizard and reboot (nb, the Splunk Forwarder doesn’t start automatically so will need starting again)
  7. Back on your main Splunk Instance, you need to now add a Receiving Port:
    1. Click Settings > Forwarding & Receiving > under Configure Receiving click Add New, and enter a port number (I used the suggested port, 9997)

Getting Data In

Right, that should be your Splunk instances up and running, and now you need some data. You can import data in many formats, but initially I’m going to focus on horst and it’s ability to export to CSV

  1. On your RPI, install horst (sudo apt-get install horst)
  2. Create a folder called SplunkFiles (mkdir /home/pi/SplunkFiles)
  3. Run cd $SPLUNK_HOME/etc/system/local
  4. Create a file called inputs.conf (sudo nano inputs.conf)
  5. Enter the following details to enable the Splunk Forwarding software to monitor:
    [default]
    host = raspberrypi
    [monitor:///home/pi/SplunkFiles/horst.csv]
    index=main
    sourcetype=csv
  6. Press ctrl-x and then y at the prompt to save
  7. Create the file outputs.conf (sudo nano outputs.conf) in the same location as inputs.conf. If the file already exists, you can overwrite it:
    [tcpout]
    defaultGroup=my_indexers
    [tcpout:my_indexers]
    # this is the ip of your Splunk Enterprise
    server=xx.xx.xx.xx:9997
  8. Restart Splunk Forwarder (run sudo ./splunk restart from $SPLUNK_HOME/bin)
  9. Set Splunk Forwarder to run at boot (if you wish) – run sudo ./splunk enable boot-start from $SPLUNK_HOME/bin
  10. Now it’s time to start Horst! First, get your interface using ifconfig
  11. Now run the following command: sudo horst -i [interface] -o /home/pi/SplunkFiles/horst.csv (eg, sudo horst -i wlx40a5ef47aa11 -o /home/pi/SplunkFiles/horst.csv)
  12. Horst should run, and all being well it will start writing to horst.csv.
  13. If everything is working, Splunk Forwarder should fire that into Splunk – thats easy to check! On your Splunk Server, click Search and Reporting and simply search for *
    Splunk Data
  14. If you don’t see data, it might be due to the date and time being wrong on your RPI – you can set this in raspi-config: https://howchoo.com/g/njnlzjmyyjg/how-to-set-the-timezone-on-your-raspberry-pi
Once you’ve got your data in, you can simplify Horst by modifying the config file: sudo nano /etc/horst.conf
Then to start horst, you just run sudo horst.


To run horst automatically, you can run sudo nano /etc/rc.local and add sudo horst & prior to exit 0

A note about data inputs:

To add additional inputs, you need to edit inputs.conf as above:

[monitor:///path/to/files/csv.csv]
index=main
sourcetype=nameofsource

You can make up the source name, but one thing I have found tricky is the ingestion of headers. There are a couple of ways to tell Splunk which headers to use, I assume there is a bug as some work some of the time, but not all, all of the time.

On your UF, you can create a file in $SPLUNK_HOME/etc/system/local called props.conf. The format of the file is as follows:

[nameofsource]
CHARSET = UTF-8
INDEXED_EXTRACTIONS = csv
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
disabled = false
FIELD_NAMES = example,header,in,csv,format

You can create the same file on your Splunk server too, each time you edit the file you will need to restart Splunk on both the forwarder and server.

An other method is to view your data as set out below, but then click Extract Fields, and follow the instructions (Delimited, Comma, etc)

Visualising Your Data

Now, I don’t intend to teach you all how to use Splunk, as it’s all really well documented, but this should give you some things to consider. Once you get used to how to handle searches, you can use them to create really cool dashboards.

To help you get started though, the easiest way to construct data is as follows

  • Always start with the source of the data, eg: host=”raspberrypi” source=”/home/pi/SplunkFiles/horst.csv”
  • A Pipe ( | ) creates fragments.. I don’t know the technical term but I just think of it as a method of separating functions
  • All fields in each event are searchable (click the chevron to view them)
    Splunk Fields
  • Here are a few examples of the easiest search types you can use to begin are:
    • timechart [latest/max/min/avg](FIELD) by FIELD
      • eg. to see the maximum SIGNAL by CHANNEL you can run timechart max(SIGNAL) by “MAC SRC”
      • You can also run multiple, such as timechart max(SIGNAL), min(SIGNAL), avg(SIGNAL) by “MAC SRC”
      • You can give friendly names to things too.. timechart min(SIGNAL) as “Min Signal” by “MAC SRC”
      • You can throw in a span to change the time range timechart span=30m min(SIGNAL) as “Min Signal” by “MAC SRC”
    • stats [latest/max/min/avg](FIELD) by FIELD
    • chart [latest/max/min/avg](FIELD) by FIELD [over FIELD]
      • chart [latest/max/min/avg](FIELD) by _time is similar to a time chart

Clicking on the Visualisation tab will allow you to display in various graph and chart type

Once you’ve got used to doing searches, you can get creative and even create dashboards

Creative Charts
Creative Charts
Splunk

Kismet

It is also possible to install Kismet and export data using the well documented Kismet API, but you need a third party app installed on Splunk which used to be free, but now costs $99,

To get it working is quite easy, you just follow the documentation on Kismet’s page as well as installing the third party app into splunk – if you use Kismet you do not need to install the Universal Forwarder.

Edit: I have very quickly thrown together a dashboard which you can copy.

Example Dashboard

In the Dashboards menu, simply click Create New Dashboard and then click on XML, you can replace the entire contents with the below – remember these are examples that I have put together very quickly, it is intended to give you a feel for how searches work and how to use dashboards:

<form>
  <label>Horst</label>
  <fieldset submitButton="false">
    <input type="time" token="field1">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="text" token="mac" searchWhenChanged="true">
      <label>MAC Address</label>
      <default>*</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Frame Type By Channel</title>
      <chart>
        <search>
          <query>host="raspberrypi" source="/home/pi/SplunkFiles/horst.csv" "WLAN TYPE"="*"
| chart count("WLAN_TYPE") over "CHANNEL" by "WLAN TYPE" usenull=false</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <refresh>10m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="charting.chart">column</option>
        <option name="charting.chart.stackMode">stacked</option>
        <option name="charting.drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
    <panel>
      <title>Top Talkers Per Channel (Bytes)</title>
      <chart>
        <search>
          <query>host="raspberrypi" source="/home/pi/SplunkFiles/horst.csv"  
| chart sum("LENGTH") as Bytes over CHANNEL by "MAC SRC"  usenull=false limit=10000</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <refresh>10m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="charting.chart">column</option>
        <option name="charting.chart.stackMode">stacked</option>
        <option name="charting.drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Avg Bytes per Client</title>
      <chart>
        <title>SSID</title>
        <search>
          <query>host="raspberrypi" source="/home/pi/SplunkFiles/horst.csv" "MAC SRC"=$mac$
| timechart span=1m avg("LENGTH") by "MAC SRC" usenull=false useother=false limit=10000</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.chart.nullValueMode">connect</option>
        <option name="charting.chart.stackMode">stacked</option>
        <option name="charting.drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
    <panel>
      <title>Avg RSSI per Client</title>
      <chart>
        <title>SSID</title>
        <search>
          <query>host="raspberrypi" source="/home/pi/SplunkFiles/horst.csv" "MAC SRC"=$mac$
| timechart span=1m avg("SIGNAL") by "MAC SRC" usenull=false useother=false limit=10000</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.chart.nullValueMode">connect</option>
        <option name="charting.chart.stackMode">stacked</option>
        <option name="charting.drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Frame Types</title>
      <chart>
        <search>
          <query>host="raspberrypi" source="/home/pi/SplunkFiles/horst.csv" "MAC SRC"=$mac$
| timechart span=1m count("WLAN TYPE") by "WLAN TYPE" usenull=false useother=false limit=10000</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="charting.chart">area</option>
        <option name="charting.chart.nullValueMode">connect</option>
        <option name="charting.chart.stackMode">stacked100</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.legend.mode">seriesCompare</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
    <panel>
      <title>Beacon RSSI</title>
      <input type="multiselect" token="SSID" searchWhenChanged="true">
        <label>SSID</label>
        <choice value="*">All</choice>
        <fieldForLabel>ESSID</fieldForLabel>
        <fieldForValue>ESSID</fieldForValue>
        <search>
          <query>host=raspberrypi source="/home/pi/SplunkFiles/horst.csv" ESSID!=ESSID
| dedup ESSID
| table ESSID</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
        <default>*</default>
      </input>
      <chart>
        <title>SSID</title>
        <search>
          <query>host="raspberrypi" source="/home/pi/SplunkFiles/horst.csv" ESSID=$SSID$ "WLAN TYPE"=BEACON
| timechart span=30s avg("SIGNAL") by "MAC SRC" usenull=false</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <refresh>1m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.chart.nullValueMode">connect</option>
        <option name="charting.chart.stackMode">stacked</option>
        <option name="charting.drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Frame Type By Channel</title>
      <chart>
        <search>
          <query>host="raspberrypi" source="/home/pi/SplunkFiles/horst.csv" "WLAN TYPE"="*"
| chart count("WLAN TYPE") over "CHANNEL" by "WLAN TYPE" limit=10000 useother=false usenull=false</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <refresh>10m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="charting.axisTitleX.visibility">collapsed</option>
        <option name="charting.axisTitleY.visibility">collapsed</option>
        <option name="charting.axisTitleY2.visibility">collapsed</option>
        <option name="charting.chart">pie</option>
        <option name="charting.chart.stackMode">stacked</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.legend.placement">none</option>
        <option name="height">487</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">1</option>
        <option name="trellis.splitBy">CHANNEL</option>
      </chart>
    </panel>
  </row>
</form>

2 thoughts on “WLAN Monitoring – Splunking on Pi

Comments are closed.