Consuming JSON with Splunk in two simple steps

Last week I needed to configure Splunk to consume JSON log fies, the documentation on the Splunk website wasn’t particularly clear, and ended in some strange results with data being repeated…With the help of an old colleague of mine (thanks Matt) he pointed me in the direction of this Splunk Answers question, which described the problem that I was having as well as the solution – fixing the configuration.
So here are the steps required to setup Splunk to consume JSON from a log file. I’ll assume that you already have an instance of Splunk installed.

Step 1 – Install the Universal Forwarder (optional)

The setup that I was working with was a Splunk server running on a Virtual Machine in Azure and an on-premise server where the log files to consume were produced. Splunk provides a useful utilities called the Universal Forwarder that consumes events data and sends it on to the Splunk server.
Installation is really straightforward so I’m not going to cover that here.

Step 2 – Configuring a custom source type

This is the part that caught me out, from the searching that I did the first time around I learnt that I needed to setup a custom source type that told Splunk to parse the data as JSON. The mistake that I made was creating this custom source type on the remote node where I had the Forwarder installed.
To do it correctly, you will need to open/create a props.conf file on the Splunk server with the following content:
[my_custom_type]
INDEXED_EXTRACTIONS = json
KV_MODE = none
 The props.conf file can be found at
$SPLUNK_HOME/etc/system/local/
If props.conf doesn’t exist in this folder (it didn’t for me) then you will need to create it.

Step 3 – Setting up log file monitoring

This is the easy part, and the part that I did do correctly, on the remote node open the inputs.conf file and add the following
[monitor://c:\logs\directory\]
sourcetype=my_custom_type
 The inputs.conf file can be found at
$SPLUNK_HOME/etc/system/local
With that done, data is going in and nothing is being duplicated.
Advertisements

2 thoughts on “Consuming JSON with Splunk in two simple steps

    • KenR says:

      Hi Peter, Splunk has a very intuitive querying language which makes it very easy to analyze data. I plan to write some future posts on how we use Splunk. But definitely worth checking out!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s