Header Ads

Monitoring IOT Device Sensor, Power Status, False Alarm using Splunk Http Event Collector


Hi people I am very grateful that you are reading me hope you are liking my work and you can reach out to me anytime you face any issue regarding things I share. I would love to help. You can reach to my content through my YouTube channel and blog of course. So this is about using Splunk tool with your existing IOT project. Splunk is a new tool people are using to monitor or visualize there processes. The processes can be an IOT setup or simple industry process that have connected plants or systems. Sorry for the PJ! Let me take a quick dive into SPLUNK.

IOT Device integration Using Splunk : 


Since I am going to use Splunk on my existing IOT project let me inform you I have shared the blogs and video on how to build an IOT Project. I hope you have heard a lot about IOT that is about connecting different devices on network to generate a cumulative smart output. These devices generate huge amount of data that needs to be stored which is stored by us on cloud storage. We are having so much data being generated everywhere around us not just because of IOT but have so many manual processes have been replaced by machines or devices which have certain parameters or constraints to be kept an eye upon. Suppose you are owner of a house and you have installed some security mechanism or say a safety system which keeps a check on amount of gases present in your house or any leakage of house hold combustible gases. This system ought to have some kind of sensors that will generate some values at some adjusted frequency (real-time in our case because disaster will not wait for specific time). These time specific values are data for us when you have something in abundant you should make the best use of it.
People these day are doing exactly this, they are playing with data so as to estimate things or make prediction to make business grow and solve problems before they could actually exist. Because at the need everything boils down to one thing is to create business and generate profit. And we need to be updated with the world. To add this analytical and monitoring feature to our IOT device we are using Splunk although there are many tools out in the market but my own favorite is Splunk.

Project Demo : 


TransScript/Cheat-Sheet :

Dashboard for monitoring IOT devices :

Problem Statement:

Usage of smart IOT devices in boom, which also increases the error occurrences in IOT devices. For E.g.: False Alarm in Fire Sensor.

Dashboard Panels:
  •         IOT device Power Status
  •          Network Status
  •          Sensor False Alarm Monitoring
  •      Realtime Sensor Data Stats


Technologies used:
  •         HEC: HTTP EVENT COLLECTOR
  •          SCRIPTED INPUT
  •          Arduino IDE
  •          PYTHON & C Programming


Whole Module Diagram:


Dashboard Outputs:

Case 1:

IOT device is Down due to Network Error:



Case 2:
Network is UP, but IOT device is Powered OFF



Case 3:
IOT device is Powered ON & Network is up, then Dashboard shows IOT sensor False Alarm (If any) and IOT sensor Realtime Data stats.



Arduino Code : 
#include <ESP8266WiFi.h> #include <ESP8266HTTPClient.h>
// Need to connec to the wifi somehow, set it up here
WiFiClient wifiClient;
String esid = "Kavin";
String epass = "1123581321";
// splunk settings and http collector token
String collectorToken = "fc87cd31-cb8a-4870-9efd-36ab953eea71";
String splunkindexer = "192.168.43.229";
String eventData="";
//you need a different client per board
String clientName ="01";
void setup()
{
//serial because seeing text telling us its working is good
Serial.begin(115200); // Reduce this if your Arduino has trouble talking so fast
Serial.println("splunk hec");
initWiFi();
Serial.print("IP address: ");
Serial.println(WiFi.localIP());
//^^ did i get an IP "its working!!"
}
void loop()
{
// build the event data, telemtry and metrics type of data goes below
String msgString ="this event is from arduino";
eventData="\"clientname\": \""+clientName + "\",\"message_recieved\": \""+String(msgString)+"\"";
Serial.println(eventData);
//send off the data
splunkpost(collectorToken,eventData,clientName,splunkindexer);
delay(10000);
// ^^ hard work, your deserve a nap
}
void initWiFi(){
Serial.println();
Serial.println("Wifi Startup");
esid.trim();
if ( esid.length() > 1 ) {
// test esid
WiFi.disconnect();
delay(100);
WiFi.mode(WIFI_STA);
Serial.print("Connecting to WiFi ");
Serial.println(esid);
WiFi.begin(esid.c_str(), epass.c_str());
if ( testWifi() == 20 ) {
return;
}
}
}
String macToStr(const uint8_t* mac)
{
String result;
for (int i = 0; i < 6; ++i) {
result += String(mac[i], 16);
if (i < 5)
result += ':';
}
return result;
}
int testWifi(void) {
int c = 0;
Serial.println("Wifi test...");
while ( c < 30 ) {
if (WiFi.status() == WL_CONNECTED) { return(20); }
delay(500);
Serial.print(".");
c++;
}
Serial.println("WiFi Connect timed out");
return(10);
}
void splunkpost(String collectorToken,String PostData, String Host, String splunkindexer)
{
// recieved the token, post data clienthost and the splunk indexer
String payload = "{ \"event\": {" + PostData + "}}";
//Build the request
HTTPClient http;
String splunkurl="http://"+ splunkindexer +":8088/services/collector/event";
String tokenValue="Splunk " + collectorToken;
// fire at will!!
http.begin(splunkurl);
http.addHeader("Content-Type", "application/json");
Serial.println(tokenValue);
http.addHeader("Authorization", tokenValue);
Serial.println(payload);
String contentlength = String(payload.length());
http.addHeader("Content-Length", contentlength );
http.POST(payload);
http.writeToStream(&Serial);
http.end();
}

 
Splunk Dashboard Code:

<form theme="light"> <label>IOT MONITOR</label>
<description>Dashboard for Monitoring IOT devices</description>
<fieldset submitButton="false"></fieldset>
<row depends="$hidden$">
<panel>
<input type="dropdown" token="new_panel" searchWhenChanged="true">
<label></label>
<choice value="$new$">$new$</choice>
<default>$new$</default>
<change>
<condition value="ON">
<set token="new_panel1">true</set>
</condition>
<condition value="OFF">
<unset token="new_panel1"></unset>
</condition>
</change>
</input>
</panel>
</row>
<row>
<panel>
<title>IOT Device Status $new|s$--$new_panel1|s$</title>
<single>
<search>
<query>index="main" | stats count(message_recieved) as data |eval newfield=if(data=0,"OFF","ON") | fields newfield</query>
<earliest>rt-1m</earliest>
<latest>rtnow</latest>
<sampleRatio>1</sampleRatio>
<progress>
<condition>
<set token="new">$result.newfield$</set>
</condition>
</progress>
</search>
<option name="colorBy">value</option>
<option name="colorMode">block</option>
<option name="drilldown">none</option>
<option name="numberPrecision">0</option>
<option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option>
<option name="rangeValues">[0,30,70,100]</option>
<option name="refresh.display">progressbar</option>
<option name="showSparkline">1</option>
<option name="showTrendIndicator">1</option>
<option name="trellis.enabled">0</option>
<option name="trellis.scales.shared">1</option>
<option name="trellis.size">medium</option>
<option name="trendColorInterpretation">standard</option>
<option name="trendDisplayMode">absolute</option>
<option name="unitPosition">after</option>
<option name="useColors">0</option>
<option name="useThousandSeparators">1</option>
</single>
</panel>
<panel depends="$new_panel1$">
<title>Sensor False Alarms</title>
<single>
<search>
<query>index="main" | head limit=0| anomalydetection |where message_recieved&gt;2000| stats count</query>
<earliest>rt-2m</earliest>
<latest>rtnow</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="colorBy">value</option>
<option name="colorMode">block</option>
<option name="drilldown">none</option>
<option name="height">173</option>
<option name="numberPrecision">0</option>
<option name="rangeColors">["0x53a051","0xf8be34","0xdc4e41"]</option>
<option name="rangeValues">[0,5]</option>
<option name="refresh.display">progressbar</option>
<option name="showSparkline">1</option>
<option name="showTrendIndicator">1</option>
<option name="trellis.enabled">0</option>
<option name="trellis.scales.shared">1</option>
<option name="trellis.size">medium</option>
<option name="trendColorInterpretation">standard</option>
<option name="trendDisplayMode">absolute</option>
<option name="underLabel">Sensor Outrage in 2 minutes</option>
<option name="unitPosition">after</option>
<option name="useColors">1</option>
<option name="useThousandSeparators">1</option>
</single>
</panel>
</row>
<row>
<panel>
<title>Network Status</title>
<single>
<search>
<query>|script print $new$ | stats last</query>
<earliest>rt-1m</earliest>
<latest>rt</latest>
<sampleRatio>1</sampleRatio>
<refresh>30s</refresh>
<refreshType>delay</refreshType>
</search>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
</single>
</panel>
<panel depends="$new_panel1$">
<title>Object distance from Sensor</title>
<single>
<search>
<query>index="main" | timechart span=2m values(message_recieved) as Distance | mvexpand Distance</query>
<earliest>rt-5m</earliest>
<latest>rt</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="colorBy">value</option>
<option name="colorMode">none</option>
<option name="drilldown">none</option>
<option name="height">147</option>
<option name="numberPrecision">0</option>
<option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option>
<option name="rangeValues">[0,30,70,100]</option>
<option name="refresh.display">progressbar</option>
<option name="showSparkline">1</option>
<option name="showTrendIndicator">1</option>
<option name="trellis.enabled">0</option>
<option name="trellis.scales.shared">1</option>
<option name="trellis.size">medium</option>
<option name="trendColorInterpretation">standard</option>
<option name="trendDisplayMode">absolute</option>
<option name="underLabel">ObstacleDistance</option>
<option name="unitPosition">after</option>
<option name="useColors">0</option>
<option name="useThousandSeparators">1</option>
</single>
</panel>
</row>
</form>



Video Transcript:

Splunk is big data tool where we going to monitor logs of our device,we are able to see the status of the IOT device if it is sending a false alarm or actually something happened and what all data it is producing. We will use http event collector of Splunk (HEC) is a fast and efficient way to send data to Splunk enterprise and Splunk Cloud ,notably it helps us to send data over https if you want more information on this and its use you can let me know and I will make another video and blog for it. So let’s start with the demo you will be able to see the splunk dashboard showing IOT Device Powered OFF which is because I haven’t connected my IOT device. I want to show an experiment if we disconnect the with the wifi network “Kavin” which is the network of my IOT device then dashboard will show a Network Error because it tries to ping the IOT device which fails due to wrong network but if I again connect to the wifi of my IOT device it shows IOT Device Powered OFF which is fine. Now we will connect our device with power its bam it shows data with multiple data frames that tells us about various things like device is ON  , sensor false alarm is 0, Device Ping Succeeded as they are on the same network then the distance of obstacle from the sensor since our IOT device is  obstacle detection System which I created for visually impaired people that uses a ultrasonic sensor for detection and Arduino UNO as processor. When I will change the distance of the obstacle the distance on the dashboard changes which clearly shows it works in real time. Splunk dashboard also shows the trends of data values of the sensor. Guys you know how important it is to make sure your system is up and working that is it is not powered off and of course it doesn’t shows up any false alarm as it will have no point having a safety system generating false alarm of such critical situations. I our case we are having an obstacle distance system but you can have any IOT system be it gas detection or fire detection or may be some temperature pressure sensors in case of plants and huge industries. So ya this is it for now if you have any problem I am saying it again and too many times please get back to me and please if you like my work don’t forget to share. Let’s stay ahead of technology. Take care guys


1 comment:

Powered by Blogger.