Top 30+ Splunk Admin Interview Questions and Answers

Last updated on 12th Nov 2021, Blog, Interview Questions

If you’re looking for Splunk Administration Interview Questions for Experienced or Freshers, you are at the right place. There are a lot of opportunities from many reputed companies in the world. So, You still have the opportunity to move ahead in your career as a Splunk Administration Analyst. we have listed some of the most important Splunk interview questions. Go through our expert-compiled questions and answers to learn about Splunk architecture, Splunk indexers and forwarders, summary index, Splunk DB Connect, transaction vs stats commands, troubleshooting Splunk performance, Splunk SDK and Web Framework, and more.

1. Compare Splunk with Spark.

Ans:

• Criteria
• Splunk
• Spark
• Deployment area
• Nature of tool
• Proprietary
• Open-source
• Working mode
• Streaming mode
• Both streaming and batch modes

2. What is Splunk?

Ans:

‘Google’ for our machine-generated data. It’s a software/engine that can be used for searching, visualizing, monitoring, reporting, etc. of our enterprise data. Splunk takes valuable machine data and turns it into powerful operational intelligence by providing real-time insights into our data through charts, alerts, reports, etc.

3. What are the common port numbers used by Splunk?

Ans:

Below are the common port numbers used by Splunk. However, we can change them if required.

Service Port Number Used :

Splunk Web port : 8000

Splunk Management port : 8089

Splunk Indexing port : 9997

Splunk Index Replication port : 8080

Splunk Network port : 514 (Used to get data from the Network port, i.e., UDP data)

KV Store : 8191

4. What are the components of Splunk? Explain Splunk architecture.?

Ans:

This is one of the most frequently asked Splunk interview questions. Below are the components of Splunk :

• Search Head: Provides the GUI for searching
• Indexer: Indexes the machine data
• Forwarder: Forwards logs to the Indexer
• Deployment Server: Manges Splunk components in a distributed environment

5. Which is the latest Splunk version in use and common port numbers used by Splunk?

Ans:

Splunk 8.2.1 (as of June 21, 2021)

6. What is Splunk Indexer? What are the stages of Splunk Indexing?

Ans:

Splunk Indexer is the Splunk Enterprise component that creates and manages indexes. The primary functions of an indexer are :

• Indexing incoming data
• Searching the indexed data
• Picture

7. What is a Splunk Forwarder? What are the types of Splunk Forwarders?

Ans:

There are two types of Splunk Forwarders as below:

• Universal Forwarder (UF)() : The Splunk agent installed on a non-Splunk system to gather data locally; it can’t parse or index data.
• Heavyweight Forwarder (HWF) : A full instance of Splunk with advanced functionalities.
It generally works as a remote collector, intermediate forwarder, and possible data filter, and since it parses data, it is not recommended for production systems.

8. Can you name a few most important configuration files in Splunk?

Ans:

• props.conf
• indexes.conf
• inputs.conf
• transforms.conf
• server.conf

9. What are the types of Splunk Licenses?

Ans:

• Enterprise license
• Free license
• Forwarder license
• Beta license
• Licenses for search heads (for distributed search
• Licenses for cluster members (for index replication

10. What is Splunk App?

Ans:

Splunk app is a container/directory of configurations, searches, dashboards, etc. in Splunk.

11. Where is Splunk Default Configuration stored?

Ans:

$splunk_home/etc/system/default

12. What are the features not available in Splunk Free?

Ans:

Splunk Free does not include below features :

• Authentication and scheduled searches/alerting
• Distributed search
• Forwarding in TCP/HTTP (to non-Splunk<)
• Deployment management

13. What happens if the License Master is unreachable?

Ans:

If the license master is not available, the license slave will start a 24-hour timer, after which the search will be blocked on the license slave (though indexing continues

14. What is the Summary Index in Splunk?

Ans:

A summary index is the default Splunk index (the index that Splunk Enterprise uses if we do not indicate another one) .

15. What is Splunk DB Connect?

Ans:

Splunk DB Connect is a generic SQL database plugin for Splunk that allows us to easily integrate database information with Splunk queries and reports.

16. Can you write down a general regular expression for extracting the IP address from logs?

Ans:

There are multiple ways in which we can extract the IP address from logs. Below are a few examples :

By using a regular expression :

• rex field=_raw “(?ip_address\d+\.\d+\.\d+\.\d+/ip_addres)”

OR

• rex field=_raw “(?ip_address([0-9]{1,3}[\.]/ip_addres. {3}[0-9]{1,3}/ip_addres) “

17. Explain Stats vs Transaction commands.

Ans:

This is another frequently asked interview question on splunk which will test Developer or Engineers knowledge. The transaction command is the most useful in two specific cases :

• When the unique ID (from one or more fields
• When an identifier is reused, say in DHCP logs, a particular message identifies the beginning or end of a transaction.
• When it is desirable to see the raw text of events combined rather than an analysis of the constituent fields of the events.

• In other cases, it’s usually better to use stats.
• As the performance of the stats command is higher, it can be used especially in a distributed search environment
• If there is a unique ID, the stats command can be used

18. How to troubleshoot Splunk performance issues??

Ans:

The answer to this question would be very wide, but mostly an interviewer would be looking for the following keywords :

• Check splunkd.log for errors
• Check server performance issues, i.e., CPU, memory usage, disk I/O, etc.
• Install the SOS (Splunk on Splunk
• Check the number of saved searches currently running and their consumption of system resources
• Install and enable Firebug, a Firefox extension. Log into Splunk (using Firefox

19. What are Buckets?

Ans:

Splunk places indexed data in directories, called ‘buckets.’ It is physically a directory containing events of a certain period.

20. What is the difference between stats and eventstats commands?

Ans:

• The stats command generates summary statistics of all the existing fields in the search results and saves them as values in new fields.
• Eventstats is similar to the stats command, except that the aggregation results are added inline to each event and only if the aggregation is pertinent to that event. The eventstats command computes requested statistics, like stats does, but aggregates them to the original raw data.

21. Who are the top direct competitors to Splunk?

Ans:

Logstash, Loggly, LogLogic, Sumo Logic, etc. are some of the top direct competitors to Splunk.

22. What do Splunk Licenses specify?

Ans:

Splunk licenses specify how much data we can index per calendar day.

23. How does Splunk determine 1 day, from a licensing perspective?

Ans:

In terms of licensing, for Splunk, 1 day is from midnight to midnight on the clock of the license master.

24. How are Forwarder Licenses purchased?

Ans:

They are included with Splunk. Therefore, there is no need to purchase it separately.

25. What is the command for restarting Splunk web server?

Ans:

This is another frequently asked Splunk commands interview question. Get a thorough idea of commands We can restart the Splunk web server by using the following command :

1. splunk start splunkweb

26. What is the command for restarting Splunk Daemon?

Ans:

Splunk Daemon can be restarted with the below command :

• splunk start splunk

27. What is the command used to check the running Splunk processes on Unix/Linux?

Ans:

If we want to check the running Splunk Enterprise processes on Unix/Linux, we can make use of the following command :

ps aux | grep splunk

28. What is the command used for enabling Splunk to boot start?

Ans:

To boot start Splunk, we have to use the following command :

$SPLUNK_HOME/bin/splunk enable boot-start

29. How to disable Splunk boot-start?

Ans:

In order to disable Splunk boot-start, we can use the following :

$SPLUNK_HOME/bin/splunk disable boot-start.

30. What is Source Type in Splunk?

Ans:

Source type is the Splunk way of identifying data.

31. What Is Stool Or How Will You Troubleshoot Splunk Configuration Files?

Ans:

Splunk tool is a command-line tool that helps us to troubleshoot configuration file issues or just see what values are being used by your Splunk Enterprise installation in an existing environment.

32. How to disable Splunk Launch Message?

Ans:

Set value OFFENSIVE=Less in splunk_launch.conf

33. How to clear Splunk Search History?

Ans:

We can clear Splunk search history by deleting the following file from Splunk server: $splunk_home/var/log/splunk/searches.log

34. What is Btool? How will you troubleshoot Splunk configuration files?

Ans:

Splunk Btool is a command-line tool that helps us troubleshoot configuration file issues or just see what values are being used by our Splunk Enterprise installation in the existing environment.

35. What is the difference between Splunk App and Splunk Add-on?

Ans:

In fact, both contain preconfigured configuration, reports, etc., but the Splunk add-on does not have a visual app. On the other hand, a Splunk app has a preconfigured visual app.

36. What are .conf files in Splunk?

Ans:

File precedence is as follows :

1. System local directory — highest priority

2. App local directories

3. App default directories

4. System default directory — lowest priority

37. What is Fishbucket? What is the Fishbucket Index?

Ans:

Fishbucket is a directory or index at the default location :

/opt/splunk/var/lib/splunk

It contains seek pointers and CRCs for the files we are indexing, so ‘splunkd’ can tell us if it has read them already. We can access it through the GUI by searching for :

index=_thefishbucket

38. How do I exclude some events from being indexed by Splunk?

Ans:

This can be done by defining a regex to match the necessary event(s

In props.conf :

[source::/var/log/foo]

# Transforms must be applied in this order

# to make sure events are dropped on the

# floor prior to making their way to the

# index processor

TRANSFORMS-set= setnull,setparsing

In transforms.conf :

[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue

[setparsing]

REGEX = login

DEST_KEY = queue

FORMAT = indexQueue

39. How can I understand when Splunk has finished indexing a log file?

Ans:

We can figure this out :

By watching data from Splunk’s metrics log in real time :

index=”_internal” source=”*metrics.log” group=”per_sourcetype_thruput” series=”<your_sourcetype_here>” |

• eval MB=kb/1024 | chart sum(MB

  By watching everything split by source type :

  index=”_internal” source=”*metrics.log” group=”per_sourcetype_thruput” | eval MB=kb/1024 | chart sum(MB

  If we are having trouble with a data input and we want a way to troubleshoot it, particularly if our whitelist/blacklist rules are not working the way we expected,

40. How to set the default search time in Splunk 6?

Ans:

To do this in Splunk Enterprise 6.0, we have to use ‘ui-prefs.conf’. If we set the value in the following, all our users would see it as the default setting :

$SPLUNK_HOME/etc/system/local

For example, if our

• $SPLUNK_HOME/etc/system/local/ui-prefs.conf file includes :
• [search]
• dispatch.earliest_time = @d
• dispatch.latest_time = now

The default time range that all users will see in the search app will be today.

41. What is the Dispatch Directory?

Ans:

$SPLUNK_HOME/var/run/splunk/dispatch contains a directory for each search that is running or has completed. For example, a directory named 1434308943.358 will contain a CSV file of its search results, a search.log with details about the search execution, and other stuff. Using the defaults (which we can override in limits.conf

42. What is the difference between Search Head Pooling and Search Head Clustering?

Ans:

Both are features provided by Splunk for the high availability of Splunk search head in case any search head goes down. However, the search head cluster is newly introduced and search head pooling will be removed in the next upcoming versions.

The search head cluster is managed by a captain, and the captain controls its slaves. The search head cluster is more reliable and efficient than the search head pooling.

43. If I want to add folder access logs from a windows machine to Splunk, how do I do it?

Ans:

Below are the steps to add folder access logs to Splunk :

• Enable Object Access Audit through group policy on the Windows machine on which the folder is located
• Enable auditing on a specific folder for which we want to monitor logs
• Install Splunk universal forwarder on the Windows machine
• Configure universal forwarder to send security logs to Splunk indexer

44. How would you handle/troubleshoot Splunk License Violation Warning?

Ans:

A license violation warning means that Splunk has indexed more data than our purchased license quota. We have to identify which index/source type has received more data recently than the usual daily data volume. We can check the Splunk license master pool-wise available quota and identify the pool for which the violation has occurred. Once we know the pool for which we are receiving more data, then we have to identify the top source type for which we are receiving more data than the usual data. Once the source type is identified, then we have to find out the source machine which is sending the huge number of logs and the root cause for the same and troubleshoot it, accordingly.

45. What is the MapReduce algorithm?

Ans:

MapReduce algorithm is the secret behind Splunk’s faster data searching. It’s an algorithm typically used for batch-based large-scale parallelization. It’s inspired by functional programming’s map(

46. How does Splunk avoid the duplicate indexing of logs?

Ans:

At the indexer, Splunk keeps track of the indexed events in a directory called fishbucket with the default location :

/opt/splunk/var/lib/splunk

It contains seek pointers and CRCs for the files we are indexing, so splunkd can tell us if it has read them already

47. What is the difference between Splunk SDK and Splunk Framework?

Ans:

Splunk SDKs are designed to allow us to develop applications from scratch and they do not require Splunk Web or any components from the Splunk App Framework.

TheSplunk App Framework resides within the Splunk web server and permits us to customize the Splunk Web UI that comes with the product and develop Splunk apps using the Splunk web server. It is an important part of the features and functionalities of Splunk, which does not license users to modify anything in Splunk.

48. For what purpose inputlookup and outputlookup are used in Splunk Search?

Ans:

The inputlookup command is used to search the contents of a lookup table. The lookup table can be a CSV lookup or a KV store lookup. The inputlookup command is considered to be an event-generating command. An event-generating command generates events or reports from one or more indexes without transforming them. There are many commands that come under the event-generating commands such as metadata, loadjob, inputcsv, etc. The inputlookup command is one of them.

Syntax :

• inputlookup [append=] [start=] [max=] [ | ] [WHERE ]

Now coming to the outputlookup command, it writes the search results to a static lookup table, or KV store collection, that we specify. The outputlookup command is not being used with external lookups.

Syntax :

• outputlookup [append=] [create_empty=] [max=] [key_field=] [createinapp=] [override_if_empty=] ( |

49. Explain how Splunk works?

Ans:

We can divide the working of Splunk into three main parts :

• Forwarder : You can see it as a dumb agent whose main task is to collect the data from various sources like remote machines and transfers it to the indexer.
• Indexer : The indexer will then process the data in real-time and store & index it on the localhost or cloud server.
• Search Head : It allows the end-user to interact with the data and perform various operations like searching, analyzing, and visualizing the information.

50. How to add the colors in Splunk UI based on the field names?

Ans:

Splunk UI has a number of features that allow the administrator to make the reports more presentable. One such feature that proves to be very useful for presenting distinguished results is the custom colors. For example, if the sales of a product drop below a threshold value, then as an administrator you can set the chart to display the values in red color.

The administrator can also change chart colors in the Splunk Web UI by editing the panels from the panel settings mentioned above the dashboard. Moreover, you can write the codes and use hexadecimal values to choose a color from the palette.

51. How the Data Ages in Splunk?

Ans:

Data entering in an indexer gets directories, also known as buckets. Over a period of time, these buckets roll over different stages from hot to warm, cold, frozen, and finally thawed. The indexer goes through a pipeline and this is where the event processing takes place. It occurs in two stages, Parsing breaks the in individual events, while indexing takes these events into the pipeline for the processing.

This is what happens to the data at each stage of the indexing pipeline :

• As soon as the data center the pipeline, it goes to the hot bucket. There can be multiple hot buckets at any point in time, which you can both search and write to.
• If any problem like the Splunk getting restarted or the hot bucket has reached a certain threshold value/size, then a new bucket will be created in its place and the existing ones roll to become a warm bucket. These warm buckets are searchable, but you cannot write anything in them.
• Further, if the indexer reaches its maximum capacity, the warm bucket will be rolled to become a cold one. Splunk will automatically execute the process by selecting the oldest warm bucket from the pipeline. However, it doesn’t rename the bucket. All the above buckets will be stored in the default location ‘$SPLUNK_HOME/var/lib/splunk/defaultdb/db/*’.
• After a certain period of time, the cold bucket rolls to become the frozen bucket. These buckets don’t have the same location as the previous buckets and are non-searchable. These buckets can either be archived or deleted based on the priorities.
• You can’t do anything if the bucket is deleted, but you can retrieve the frozen bucket if it’s being archived. The process of retrieving an archived bucket is known as thawing. Once a bucket is thawed it becomes searchable and stores into a new location ‘$SPLUNK_HOME/var/lib/splunk/defaultdb/thaweddb/’.

52. What are pivots and data models in Splunk?

Ans:

Data models in Splunk are used when you have to process huge amounts of unstructured data and create a hierarchical model without executing complex search queries on the data. Data models are widely used for creating sales reports, add access levels, and create a structure of authentication for various applications.

Pivots, on the other hand, give you the flexibility to create multiple views and see the results as per the requirements. With pivots, even the managers of stakeholders from non-technical backgrounds can create views and get more details about their departments.

53. Explain Workflow Actions?

Ans:

This topic will be present in any set of Splunk interview questions and answers. Workflow actions in Splunk are referred to as highly configurable, knowledge objects that enable you to interact with web resources and other fields. Splunk workflow actions can be used to create HTML links and use them to search field values, put HTTP post requests for specific URLs, and run secondary searches for selected events.

54. How many types of dashboards are available in Splunk?

Ans:

There are three types of dashboards available in Splunk :

• Real-time dashboards
• Dynamic form-based dashboards
• Dashboards for scheduled reports

55. What are the types of alerts available in Splunk?

Ans:

Alerts are the actions generated by a saved search result after a certain period of time. Once an alert has occurred, subsequent actions like email or message will also be triggered. There two Types of alters available in Splunk :

• Real-time alerts : we can divide the real-time alerts into two parts, pre-result, and rolling-window alerts. The pre-result alert gets triggered with every search, while rolling-window alerts are triggered when a specific criterion is met by the search.
• Scheduled Alerts : As the name suggests, scheduled alerts can be initialized to trigger multiple alerts based on the set criteria.

56. Define the term “Search factor” and “Replication factor”

Ans:

Search factor : The search factor (SF

Replication factor : The replication factor (RF

57. How to stop/start the Splunk service?

Ans:

The command for starting Splunk service :

./splunk start

The command for stopping Splunk service :

./splunk stop

58. What is the use of Time Zone property in Splunk?

Ans:

Time Zone is an important property that helps you search for the events in case any fraud or security issue occurs. The default time zone will be taken from the browser settings or the machine you are using. Apart from event searching, it is also used in data pouring from multiple sources and aligns them based on different time zones.

59. What are the important Search commands in Splunk?

Ans:

Below are some of the important search commands in Splunk :

• Erex
• Abstract
• Typer
• Rename
• Anomalies
• Fill down
• Accum
• Add totals

60. How many types of search modes are there in Splunk?

Ans:

There are three types of search modes in Splunk :

• Fast mode : speeds up your search result by limiting the types of data.
• Verbose mode : Slower as compared to the fast mode, but returns the information for as many events as possible.
• Smart mode : It toggles between different modes and search behaviours to provide maximum results in the shortest period of time.

61. Why is Splunk used for analyzing machine data?

Ans:

This question will most likely be the first question you will be asked in any Splunk interview. You need to start by saying that :

Splunk is a platform which allows people to get visibility into machine data that is generated from hardware devices, networks, servers, IoT devices and other sources.

Splunk is used for analyzing machine data because of following reasons :

Business Insights :

Splunk understands the trends, patterns and then gains the operational intelligence from the machine data which in turn helps in taking better informed business decisions.

Operational Visibility :

Using the machine data Splunk obtains an end-to-end visibility across operations and then breaks it down across the infrastructure.

Proactive Monitoring :

Splunk uses the machine data to monitor systems in the real time which helps in identifying the issues, problems and even attacks.

Search & Investigation :

Machine data is also used to find and fix the problems, correlate events across multiple data sources and implicitly detect patterns across massive sets of data by Splunk.

62. Why use only Splunk? Why can’t I go for something that is open source?

Ans:

This kind of question is asked to understand the scope of your knowledge. You can answer that question by saying that Splunk has a lot of competition in the market for analyzing machine logs, doing business intelligence, for performing IT operations and providing security. But, there is no one single tool other than Splunk that can do all of these operations and that is where Splunk comes out of the box and makes a difference. With Splunk you can easily scale up your infrastructure and get professional support from a company backing the platform. Some of its competitors are Sumo Logic in the cloud space of log management and ELK in the open source category. You can refer to the below table to understand how Splunk fares against other popular tools feature-wise.

63. Which Splunk Roles can share the same machine?

Ans:

This is another frequently asked Splunk interview question which will test the candidate’s hands-on knowledge. In case of small deployments, most of the roles can be shared on the same machine which includes Indexer, Search Head and License Master. However, in case of larger deployments the preferred practice is to host each role on stand alone hosts. Details about roles that can be shared even in case of larger deployments are mentioned below :

• Strategically, Indexers and Search Heads should have physically dedicated machines. Using Virtual Machines for running the instances separately is not the solution because there are certain guidelines that need to be followed for using computer resources and spinning multiple virtual machines on the same physical hardware can cause performance degradation.
• However, a License master and Deployment server can be implemented on the same virtual box, in the same instance by spinning different Virtual machines.
• You can spin another virtual machine on the same instance for hosting the Cluster master as long as the Deployment master is not hosted on a parallel virtual machine on that same instance because the number of connections coming to the Deployment server will be very high.
• This is because the Deployment server not only caters to the requests coming from the Deployment master, but also to the requests coming from the Forwarders.

64. What are the unique benefits of getting data into a Splunk instance via Forwarders?

Ans:

You can say that the benefits of getting data into Splunk via forwarders are bandwidth throttling, TCP connection and an encrypted SSL connection for transferring data from a forwarder to an indexer. The data forwarded to the indexer is also load balanced by default and even if one indexer is down due to network outage or maintenance purpose, that data can always be routed to another indexer instance in a very short time. Also, the forwarder caches the events locally before forwarding it, thus creating a temporary backup of that data.

65. Briefly explain the Splunk Architecture

Ans:

Look at the below image which gives a consolidated view of the architecture of Splunk. You can find the detailed explanation in this link

66. Give a few use cases of Knowledge objects.

Ans:

Knowledge objects can be used in many domains. Few examples Physical Security: If your organization deals with physical security, then you can leverage data containing information about earthquakes, volcanoes, flooding, etc to gain valuable insights

Application Monitoring : By using knowledge objects, you can monitor your applications in real-time and configure alerts which will notify you when your application crashes or any downtime occurs

Network Security : You can increase security in your systems by blacklisting certain IPs from getting into your network. This can be done by using the Knowledge object called lookups.

Employee Management : If you want to monitor the activity of people who are serving their notice period, then you can create a list of those people and create a rule preventing them from copying data and using them outside

Easier Searching Of Data : With knowledge objects, you can tag information, create event types and create search constraints right at the start and shorten them so that they are easy to remember, correlate and understand rather than writing long search queries. Those constraints where you put your search conditions, and shorten them are called event types.

These are some of the operations that can be done from a non-technical perspective by using knowledge objects. Knowledge objects are the actual application in business, which means Splunk interview questions are incomplete without Knowledge objects. In case you want to read more about the different knowledge objects available and how they can be used, read this blog

67. Why should we use Splunk Alert? What are the different options while setting up Alerts?

Ans:

This is a common question aimed at candidates appearing for the role of a Splunk Administrator. Alerts can be used when you want to be notified of an erroneous condition in your system. For example, send an email notification to the admin when there are more than three failed login attempts in a twenty-four hour period. Another example is when you want to run the same search query every day at a specific time to give a notification about the system status. Different options that are available while setting up alerts are :

• You can create a web hook, so that you can write to hipchat or github. Here, you can write an email to a group of machines with all your subject, priorities, and body of the message
• You can add results, .csv or pdf or inline with the body of the message to make sure that the recipient understands where this alert has been fired, at what conditions and what is the action he has taken
• You can also create tickets and throttle alerts based on certain conditions like a machine name or an IP address. For example, if there is a virus outbreak, you do not want every alert to be triggered because it will lead to many tickets being created in your system which will be an overload. You can control such alerts from the alert window.

68. Explain Data Models and Pivot

Ans:

Data models are used for creating a structured hierarchical model of your data. It can be used when you have a large amount of unstructured data, and when you want to make use of that information without using complex search queries.

A few use cases of Data models are :

• Create Sales Reports: If you have a sales report, then you can easily create the total number of successful purchases, below that you can create a child object containing the list of failed purchases and other views
• Set Access Levels: If you want a structured view of users and their various access levels, you can use a data model
• Enable Authentication: If you want structure in the authentication, you can create a model around VPN, root access, admin access, non-root admin access, authentication on various different applications to create a structure around it in a way that normalizes the way you look at data. So when you look at a data model called authentication, it will not matter to Splunk what the source is, and from a user perspective it becomes extremely simple because as and when new data sources are added or when old one’s are deprecated, you do not have to rewrite all your searches and that is the biggest benefit of using data models and pivots.

On the other hand with pivots, you have the flexibility to create the front views of your results and then pick and choose the most appropriate filter for a better view of results. Both these options are useful for managers from a non-technical or semi-technical background.

69. Explain Search Factor (SF. & Replication Factor (RF.

Ans:

Questions regarding Search Factor and Replication Factor are most likely asked when you are interviewing for the role of a Splunk Architect. SF & RF are terminologies related to Clustering techniques (Search head clustering & Indexer clustering)

• The search factor determines the number of searchable copies of data maintained by the indexer cluster. The default value of the search factor is 2. However, the Replication Factor in case of Indexer cluster, is the number of copies of data the cluster maintains and in case of a search head cluster, it is the minimum number of copies of each search artifact, the cluster maintains
• Search head cluster has only a Search Factor whereas an Indexer cluster has both a Search Factor and a Replication Factor
• Important point to note is that the search factor must be less than or equal to the replication factor

70. Which commands are included in the ‘filtering results’ category?

Ans:

There will be a great deal of events coming to Splunk in a short time. Thus it is a little complicated task to search and filter data. But, thankfully there are commands like ‘search’, ‘where’, ‘sort’ and ‘rex’ that come to the rescue. That is why, filtering commands are also among the most commonly asked Splunk interview questions.

Search : The ‘search’ command is used to retrieve events from indexes or filter the results of a previous search command in the pipeline. You can retrieve events from your indexes using keywords, quoted phrases, wildcards, and key/value expressions. The ‘search’ command is implied at the beginning of any and every search operation.

Where : The ‘where’ command however uses ‘eval’ expressions to filter search results. While the ‘search’ command keeps only the results for which the evaluation was successful, the ‘where’ command is used to drill down further into those search results. For example, a ‘search’ can be used to find the total number of nodes that are active but it is the ‘where’ command which will return a matching condition of an active node which is running a particular application.

Sort : The ‘sort’ command is used to sort the results by specified fields. It can sort the results in a reverse order, ascending or descending order. Apart from that, the sort command also has the capability to limit the results while sorting. For example, you can execute commands which will return only the top 5 revenue generating products in your business.

Rex : The ‘rex’ command basically allows you to extract data or particular fields from your events. For example if you want to identify certain fields in an email id: abc@edureka.co, the ‘rex’ command allows you to break down the results as abc being the user id, edureka.co being the domain name and edureka as the company name. You can use rex to breakdown, slice your events and parts of each of your events record the way you want.

71. What is a lookup command? Differentiate between inputlookup & outputlookup commands. Lookup command is that topic into which most interview questions dive into, with questions like: Can you enrich the data? How do you enrich the raw data with external lookup?

Ans:

You will be given a use case scenario, where you have a csv file and you are asked to do lookups for certain product catalogs and asked to compare the raw data & structured csv or json data. So you should be prepared to answer such questions confidently.

72. What is the difference between ‘eval’, ‘stats’, ‘charts’ and ‘timecharts’ commands?

Ans:

‘Eval’ and ‘stats’ are among the most common as well as the most important commands within the Splunk SPL language and they are used interchangeably in the same way as ‘search’ and ‘where’ commands.

• At times ‘eval’ and ‘stats’ are used interchangeably however, there is a subtle difference between the two. While the ‘stats‘ command is used for computing statistics on a set of events, ‘eval’ command allows you to create a new field altogether and then use that field in subsequent parts for searching the data.
• Another frequently asked question is the difference between ‘stats’, ‘charts’ and ‘timecharts’ commands. The difference between them is mentioned in the table below.

1. Stats vs Chart vs TimeChart

2. Stats

3. Chart

4. Timechart

5. Stats is a reporting command which is used to present data in a tabular format.

73. What are the different types of Data Inputs in Splunk?

Ans:

This is the kind of question which only somebody who has worked as a Splunk administrator can answer. The answer to the question is below.

• The obvious and the easiest way would be by using files and directories as input
• Configuring Network ports to receive inputs automatically and writing scripts such that the output of these scripts is pushed into Splunk is another common way
• But a seasoned Splunk administrator would be expected to add another option called windows inputs. These windows inputs are of 4 types: registry inputs monitor, printer monitor, network monitor and active directory monitor.

74. What are the default fields for every event in Splunk?

Ans:

There are about 5 fields that are default and they are barcoded with every event into Splunk. They are host, source, source type, index and timestamp.

75. Explain file precedence in Splunk.

Ans:

File precedence is an important aspect of troubleshooting in Splunk for an administrator, developer, as well as an architect. All of Splunk’s configurations are written within plain text .conf files. There can be multiple copies present for each of these files, and thus it is important to know the role these files play when a Splunk instance is running or restarted. File precedence is an important concept to understand for a number of reasons :

1. To be able to plan Splunk upgrades

2. To be able to plan app upgrades

3. To be able to provide different data inputs

4. To distribute the configurations to your splunk deployments.

5. To determine the priority among copies of a configuration file, Splunk software first determines the directory scheme. The directory schemes are either a Global or b App/user.

When the context is global (that is, where there’s no app/user context) directory priority descends in this order :

• System local directory — highest priority
• App local directories
• App default directories
• System default directory — lowest priority

When the context is app/user, directory priority descends from user to app to system :

• User directories for current user — highest priority
• App directories for currently running app (local, followed by default
• App directories for all other apps (local, followed by default
• System directories (local, followed by default

76. How can we extract fields?

Ans:

You can extract fields from either event lists, sidebar or from the settings menu via the UI.

The other way is to write your own regular expressions in the props.conf configuration file.

77. What is the difference between Search time and Index time field extractions?

Ans:

As the name suggests, Search time field extraction refers to the fields extracted while performing searches whereas, fields extracted when the data comes to the indexer are referred to as Index time field extraction. You can set up the indexer time field extraction either at the forwarder level or at the indexer level.

Another difference is that Search time field extraction’s extracted fields are not part of the metadata, so they do not consume disk space. Whereas index time field extraction’s extracted fields are a part of metadata and hence consume disk space.

78. How does Splunk help in the Organization?

Ans:

Most of the corporations are investing in this technology as it helps to examine their end-to-end infrastructures, shun service outages & gain real-time critical insights into client experience, key business metrics & transactions.

79. What is the summary index in Splunk?

Ans:

Summary index is another important Splunk interview question from an administrative perspective. You will be asked this question to find out if you know how to store your analytical data, reports and summaries. The answer to this question is below.

The biggest advantage of having a summary index is that you can retain the analytics and reports even after your data has aged out. For example :

• Assume that your data retention policy is only for 6 months but, your data has aged out and is older than a few months. If you still want to do your own calculation or dig out some statistical value, then during that time, summary index is useful
• For example, you can store the summary and statistics of the percentage growth of sales that took place in each of the last 6 months and you can pull the average revenue from that. That average value is stored inside the summary index.

But the limitations with summary index are :

• You cannot do a needle in the haystack kind of a search
• You cannot drill down and find out which products contributed to the revenue
• You cannot find out the top product from your statistics
• You cannot drill down and nail which was the maximum contribution to that summary.

That is the use of Summary indexing and in an interview, you are expected to answer both these aspects of benefit and limitation.

80. How to exclude some events from being indexed by Splunk?

Ans:

You might not want to index all your events in a Splunk instance. In that case, how will you exclude the entry of events to Splunk.

An example of this is the debug messages in your application development cycle. You can exclude such debug messages by putting those events in the null queue. These null queues are put into transforms.conf at the forwarder level itself.

If a candidate can answer this question, then he is most likely to get hired.

81. What is the use of Time Zone property in Splunk? When is it required the most?

Ans:

Time zone is extremely important when you are searching for events from a security or fraud perspective. If you search your events with the wrong time zone then you will end up not being able to find that particular event altogether. Splunk picks up the default time zone from your browser settings. The browser in turn picks up the current time zone from the machine you are using. Splunk picks up that timezone when the data is input, and it is required the most when you are searching and correlating data coming from different sources. For example, you can search for events that came in at 4:00 PM IST, in your London data center or Singapore data center and so on. The timezone property is thus very important to correlate such events.

82. How to assign colors in a chart based on field names in Splunk UI? You need to assign colors to charts while creating reports and presenting results. But what if you want to assign your own colors?

Ans:

For example, if your sales numbers fall below a threshold, then you might need that chart to display the graph in red color. Then, how will you be able to change the color in a Splunk Web UI.

You will have to first edit the panels built on top of a dashboard and then modify the panel settings from the UI. You can then pick and choose the colors. You can also write commands to choose the colors from a palette by inputting hexadecimal values or by writing code. But, Splunk UI is the preferred way because you have the flexibility to assign colors easily to different values based on their types in the bar chart or line chart. You can also give different gradients and set your values into a radial gauge or water gauge.

83. What is sourcetype in Splunk?

Ans:

Now this question may feature at the bottom of the list, but that doesn’t mean it is the least important among other Splunk interview questions.

Sourcetype is a default field which is used to identify the data structure of an incoming event. Sourcetype determines how Splunk Enterprise formats the data during the indexing process. Source type can be set at the forwarder level for indexer extraction to identify different data formats. Because the source type controls how Splunk software formats incoming data, it is important that you assign the correct source type to your data. It is important that even the indexed version of the data (the event data) also looks the way you want, with appropriate timestamps and event breaks. This facilitates easier searching of data later.

For example, the data may come in the form of a csv, such that the first line is a header, the second line is a blank line and then from the next line comes the actual data. Another example where you need to use sourcetype is if you want to break down the date field into 3 different columns of a csv, each for day, month, year and then index it.

84. What are alerts in Splunk?

Ans:

An alert is an action that a saved search triggers on regular intervals set over a time range, based on the results of the search. When the alerts are triggered, various actions occur consequently. For instance, sending an email when a search to the predefined list of people is triggered.

85. What are the types of alerts in splunk?

Ans:

• Pre-result alerts : Most commonly used alert type and runs in real-time for an all-time span. These alerts are designed such that whenever a search returns a result, they are triggered.
• Scheduled alerts : The second most common- scheduled results are set up to evaluate the results of a historical search result running over a set time range on a regular schedule. You can define a time range, schedule and the trigger condition to an alert.
• Rolling-window alerts : These are the hybrid of pre-result and scheduled alerts. Similar to the former, these are based on real-time search but do not trigger each time the search returns a matching result. It examines all events in real-time mapping within the rolling window and triggers the time that specific condition by that event in the window is met like the scheduled alert is triggered on a scheduled search.

86. Explain the bucket lifecycle?

Ans:

A directory that contains indexed data is known as a Splunk bucket. It also contains events of a certain period. Bucket lifecycle includes the following stages :

• Hot – It contains newly indexed data and is open for writing. For each index, there are one or more hot buckets available
• Warm – Data rolled from hot
• Cold – Data rolled from warm
• Frozen – Data rolled from cold. The indexer deletes frozen data by default but users can also archive it.
• Thawed – Data restored from an archive. If you archive frozen data, you can later return it to the index by thawing.

87. What is the eval command in splunk?

Ans:

It evaluates an expression and consigns the resulting value into a destination field. If the destination field matches with an already existing field name, the existing field is overwritten with the eval expression. This command evaluates Boolean, mathematical and string expressions.

88. Uses of eval command in splunk?

Ans:

• Convert Values
• Round Values
• Perform Calculations
• User conditional statements
• Format Values

89. Explain the difference between search head pooling and search head clustering?

Ans:

Search head pooling is a group of connected servers that are used to share the load, Configuration and user data Whereas Search head clustering is a group of Splunk Enterprise search heads used to serve as a central resource for searching. Since the search head cluster supports member interchangeability, the same searches and dashboards can be run and viewed from any member of the cluster.

90. Explain the function of Alert Manager?

Ans:

Alert manager displays the list of most recently fired alerts, i.e. alert instances. It provides a link to view the search results from that triggered alert. It also displays the alert’s name, app, type (scheduled, real-time, or rolling window

91. What is SOS?

Ans:

SOS stands for Splunk on Splunk. It is a Splunk app that provides a graphical view of your Splunk environment performance and issues. It has the following purposes :

• Diagnostic tool to analyze and troubleshoot problems
• Examine Splunk environment performance
• Solve indexing performance issues
• Observe scheduler activities and issues
• See the details of the scheduler and user-driven search activity
• Search, view and compare configuration files of Splunk

92. Who Are The Biggest Direct Competitors To Splunk?

Ans:

• logstash
• Logged
• Loglogic
• sumo logic etc..

93. What is the difference between the Splunk App Framework and Splunk SDKs?

Ans:

Splunk App Framework resides within Splunk’s web server and permits you to customize the Splunk Web UI that comes with the product and develop Splunk apps using the Splunk web server. It is an important part of the features and functionalities of Splunk Software, which does not license users to modify anything in the Splunk Software.

Splunk SDKs are designed to allow you to develop applications from the ground up and not require Splunk Web or any components from the Splunk App Framework. These are separately licensed to you from the Splunk Software and do not alter the Splunk Software.

94. What is Splunk indexer and explain its stages?

Ans:

The indexer is a Splunk Enterprise component that creates and manages indexes. The main functions of an indexer are :

• Indexing incoming data
• Searching indexed data

Splunk indexer has the following stages :

Input : Splunk Enterprise acquires the raw data from various input sources and breaks it into 64K blocks and assigns them some metadata keys.

These keys include host, source and source type of the data.

• Parsing : Also known as event processing, during this stage, the Enterprise analyzes and transforms the data, breaks data into streams, identifies, parses and sets timestamps, performs metadata annotation and transformation of data.
• Indexing : In this phase, the parsed events are written on the disk index including both compressed data and the associated index files.
• Searching : The ‘Search’ function plays a major role during this phase as it handles all searching aspects (interactive, scheduled searches, reports, dashboards, alerts on the indexed data and stores saved searches, events, field extractions and views.

95. List .conf files by priority?

Ans:

File precedence in Splunk is as follows :

• System local directory: top priority
• App local directories
• App default directories
• System default directory : lowest priority

96. Where is Splunk default configuration stored?

Ans:

Splunk default configuration is stored at $splunk_home/etc/system/default

97. Give a few use cases of Knowledge Objects.

Ans:

Knowledge objects can be used in many domains. Few examples are :

• Application Monitoring: Your applications can be monitored in real-time with configured alerts to notify when an application crashes.
• Physical Security: You can have the full leverage of the data containing information about the volcanos, floods, etc. to gain insights, if your firm deals with them.
• Network Security: With the usage of lockups from your knowledge objects, you can increase security in your systems by blacklisting certain IPs from getting into your network.
• Employee Management: If you want to monitor the activity of people who are serving their notice period, then you can create a list of those people and create a rule preventing them from copying data and using them outside.

98. How to list all the saved searches in Splunk?

Ans:

Using syntax :

rest /servicesNS/-/-/saved/searches splunk_server=loca

99. Which Is the Latest Splunk Version In Use?

Ans:

Splunk 6.3.

100. What Is Stool Or How Will You Troubleshoot Splunk Configuration Files?

Ans:

Splunk tool is a command-line tool that helps us to troubleshoot configuration file issues or just see what values are being used by your Splunk Enterprise installation in an existing environment.