Splunk local threat intel

Why yet another Splunk local threat intel article ? Obviously, there are many excellent articles:

• Threat Intelligence framework in Splunk ES
• Dissecting the Threat Intelligence Framework
• Threat Intel and Splunk Enterprise Security Part 1
• Threat Intel and Splunk Enterprise Security Part 2

Unlike other articles, we mainly focus on common operational issues of Splunk local threat intel usage including:

• Understand the impact of editing Splunk local threat intel csv lookup
• Removing Splunk local threat intel entry

Before we start to discuss those operational issues, let’s explore the workflow of threat intelligence framework. Basically, it consists of 4 phases:

1. Threat intelligence manager script firstly downloaded raw data. If it is threat related feed, manager script will further normalizes and stores it to different KV Store inside the Threat Collections.
2. Multiple Threat “Lookup Gen” search transform non-threat raw files or threat collections KV Store to different csv files.
3. Multiple “Threat Gen” scheduled search running tstats command to check matching values between output csv files from step 2 and different data model. In case “Threat Gen” search find a matching value, it will output to threat_activity index.
4. Schedule “Threat Activity Detected” correlation search look for events in threat_activity index from step 3 and then generate notable events.

Threat intelligence manager

In short, Threat intelligence manager is a python script located at DA-ESS-ThreatIntelligence/bin/threat_intelligence_manager.py.

Manager Stage 1 – download feeds

Firstly, manager will download feeds (including local threat intel) defined in inputs.conf based on the configured interval and store it into $SPLUNK_DB/modinputs/threatlist directory. As a result of Splunk Enterprise Security app installation, there are 2 different inputs.conf which can be located in DA-ESS-ThreatIntelligence 及 SA-ThreatIntelligence directory.

Now, let’s take a look of default threat intelligence feed by navigating Enterprise Security -> Configure -> Data Enrichment-> Intelligence Downloads. Some of these feeds are disabled by default.

Then, we will further check the detail configuration page, and notice the “Is Threat Intelligence” option. If this option is enabled, Splunk will further process the feed and save into Threat collection KV store. In fact, KV Store is key-value pair stored in MongoDB, which can be retrieved via inputlookup command.

By default, Splunk Enterprise Security comes with 2 main feed types:

• Non-Threat Intelligence feeds (Log enrichment feeds or other feeds useful for use case development and investigation inside SA-ThreatIntelligence directory):
  • cisco_top_one_million_sites (Cisco Umbrella 1 Million)
  • alexa_top_one_million_sites (retired and no maintenance)
  • MaxMind GeoIP ASN IPv4/IPv6 database (Only support old database format)
  • icann_top_level_domain_list

• Threat Intelligence feeds (inside DA-ESS-ThreatIntelligence directory)
  • Common open source threat feeds including phishtank, zeus blacklists
  • Local threat intel csv file
    • local_certificate_intel.csv
    • local_domain_intel.csv
    • local_email_intel.csv
    • local_file_intel.csv
    • local_http_intel.csv
    • local_ip_intel.csv
    • local_process_intel.csv
    • local_registry_intel.csv
    • local_service_intel.csv
    • local_user_intel.csv

Splunk now comes with both alexa_top_one_million_sites 及 cisco_top_one_million_sites. In fact, Splunk is moving to Cisco umbrella top 1 million feed as its default. However, we noticed some knowledge objects still reference to the OLD Alexa feed. Therefore, we recommended to replace the URL of Alexa with cisco umbrella top 1 million as 2 feeds are basically compatible.

Manager Stage 2 – normalize threat feeds

Secondly, the script will normalize the downloaded raw file and stored it in different KV Store collections based on the field name if it is classified as “Is Threat Intelligence“. Those field name mapping are available in ./DA-ESS-ThreatIntelligence/default/collections.conf. Below is a snippet for http_intel threat collection. Basically, if the field name inside threat intel feed matches the below field name (e.g. http_version, http_method), Splunk will save it into http_intel KV store.

[http_intel]
field.http_version              = string
field.http_method               = string
field.http_content_type         = string
field.http_referrer             = string
field.http_user_agent           = string
field.http_user_agent_length    = number
field.status                    = number
field.cookie                    = string
field.header                    = string
field.data                      = string
field.url                       = string
field.url_length                = number
field.uri_path                  = string
field.uri_query                 = string
field.ip                        = string
field.domain                    = string
field.description               = string
field.threat_key                = string
field.time                      = time
field.weight                    = number
field.disabled                  = bool
field.updated                   = bool

Furthermore, there is an interesting field “domain” defined in http_intel, ip_intel and certificate_intel collection. If we manually add a column named domain to the local_http_intel.csv file, Splunk will still process it. However, you will likely see the domain intel added to the ip_intel lookup instead of http_intel due to de-duplication process by manager script.

Another thing worth to mention is “disabled” and “updated” field. Splunk will not process it even if you manually add these 2 columns.

Below list threat related KV Store collections in collections.conf:

• http_intel
• ip_intel
• certificate_intel
• email_intel
• process_intel
• file_intel
• registry_intel
• service_intel
• user_intel

Scenario 1 – Added wrong Splunk local threat intel

Let’s take an example editing local_domain_intel.csv. In our example, we use the Splunk lookup editor app available 網址. Notice that we accidentally add “gooddomain.com” into local_domain_intel csv file.

After several minutes, Splunk process local_domain_intel.csv and add those entries into ip_intel KV Store.

Next, we removed the entry gooddomain.com.

When we back to check the ip_intel KV Store, gooddomain.com is still there. I have actually seen some junior SOC analyst added the entire CDN domain into domain threat intel.

In normal case, I think nobody always check the ip_intel, and you always discover this when “Threat Activity Detected” filling up the “Incident Review” page.

One of solution to this situation is removing ALL local_domain_intel from ip_intel Threat Collection by using the following query. Then, we can remove the entry gooddomain.com in local_domain_intel.csv.

| inputlookup ip_intel
| search NOT threat_key="local_domain_intel"
| outputlookup ip_intel

Scenario 2 – Using asterisk in Splunk local threat intel

Let’s explore the local_http_intel.csv example. As a Splunker, you may think that first 2 entry make sense to you, but it may not work as expected. We will cover the effect of each entry in the following section.

Threat “Lookup Gen” search

Splunk Enterprise Security comes with 2 types of lookup generation search, which is non-threat related (SA-ThreatIntelligence) and threat related (DA-ESS-ThreatIntelligence). All these lookup gen search aimed to transform the source files to csv files used by other Splunk search. In addition, all these search with prefix “Threat” and suffix “Lookup Gen”.

Non-threat related lookup gen

We will take a look of below non-threat related lookup generation.

Let’s further check the query of “Threat – Alexa Top Sites – Lookup Gen“.

| inputintelligence `top_1m_sites` | outputlookup alexa_lookup_by_str | stats count

In the above query, we can see top_1m_sites enclosed with symbol `, and it is a Splunk macro. We can check the query embedded inside the macro by navigating to Settings -> Advanced search -> Search macros.

The top_1m_sites macro actually refers to cisco_top_one_million_sites. As I have said, non-threat related feeds are stored in $SPLUNK_DB/modinputs/threatlist directory without any transformation. This inputintelligence command reads the raw file cisco_top_one_million_sites inside $SPLUNK_DB/modinputs/threatlist directory and output somewhere. In addition, you may also notice the keyword alexa is still referenced by Splunk.

Threat related lookup gen

Next, we will move on to threat related lookup gen. As there are a lots threat related lookup gen, thus we select HTTP related threat lookup generation search to facilitate our discussion. One important thing is that ALL enabled “Lookup Gen” will still keep running even “Next Scheduled Time” is “none”.

In order to explain the example mentioned in previous section, we selected 4 lookup gen in our discussion:

Threat - Threat Intelligence By HTTP User Agent - Lookup Gen
Threat - Threat Intelligence By HTTP User Agent Wildcard - Lookup Gen
Threat - Threat Intelligence By HTTP User Agent Wildcard - Lookup Gen
Threat - Threat Intelligence By URL - Lookup Gen

Threat – Threat Intelligence By HTTP User Agent – Lookup Gen

Firstly, we will take a look of the base query of this lookup gen. As we can see, it contains 3 macros.

| `http_intel` | `exclude_disabled_entries` | `threatintel_outputlookup(http_user_agent)`

Let’s explore the mentioned 3 macro starting with `http_intel` first. Basically, it runs inputlookup http_intel and enrich some more fields.

inputlookup append=T http_intel | fillnull value=0 updated,disabled | `set_threat_collection_name("http_intel")` | `get_threat_attribution_weight(threat_key)`

For the `exclude_disabled_entries` macro, it simply filters all row with value disabled=1.

where NOT disabled=1

Finally, `threatintel_outputlookup(http_user_agent)` is similar to | fields http_user_agent outputlookup threatintel_by_http_user_agent.

`threatintel_outputlookup("$field$","threatintel_by_$field$")`

So, how is the resulting csv threatintel_by_http_user_agent looks like? As we can seen below, the lookup do not contains other fields such as url and http_referrer, but only the field http_user_agent.

Threat – Threat Intelligence By HTTP User Agent Wildcard – Lookup Gen

Secondly, we will check another lookup gen with a very close name. The only difference is it contains “Wildcard” keyword. The real difference behind is the final output destination is threatintel_by_http_user_agent_wildcard.

| `http_intel` | `exclude_disabled_entries` | `threatintel_outputlookup_wildcard(http_user_agent)`

Please note that the below threatintel_by_http_user_agent_wildcard.csv is screen is captured from Splunk Enterprise Security v5.3.1. As shown below, a wildcard appears in http_user_agent field as a result of row 1 in Scenario 2. At this point, I think every Splunker aware of the issue, which is notable event will be generated whenever http_user_agent is not empty. This issue is fixed since Splunk Enterprise Security v6.0, and field value with ONLY asterisk will not be populated to wildcard lookup table.

Now, we can conclude 2 things of Splunk Threat intel framework:

• Field value with wildcard and without wildcard are separated into 2 different lookup table
• Each field (e.g. http_user_agent, url) are distributed into different lookup table for further processing. In other words, the notable event will be generated because of a single field value, but not the combination of multiple field values.

Threat – Threat Intelligence By URL – Lookup Gen

This lookup gen contains one additional macro `mvappend_field(url,http_referrer)`, which effectively rename ALL http_referrer inside threat feed to url field.

| `http_intel` | `exclude_disabled_entries` | `mvappend_field(url,http_referrer)` | `threatintel_outputlookup(url)`

If we check the resulting threatintel_by_url.csv, it contains only one entry. Therefore, do not expect any notable event due to http_referrer in raw log match the value in threat intel.

Threat – Threat Intelligence By URL Wildcard – Lookup Gen

If you follow all the above 3 lookup gen section, you will understand how the following query works.

| `http_intel` | `exclude_disabled_entries` | `mvappend_field(url,http_referrer)` | `threatintel_outputlookup_wildcard(url)`

One thing worth to discuss here is I usually recommended those SOC analyst to add wildcard to url field after the last characters so that we will not miss anything. Otherwise, splunk will only match the exact URL.

“Threat Gen” search

In this section, we will cover “Threat Gen” search. In short, they are scheduled search with prefix “Threat” and suffix “Threat Gen” running tstats command to match threatintel csv lookup. You must explicit specific a data model when using tstats command.

Splunk Data Model

Many SOC analyst comes to me and ask what is data model ? In short, Data Model groups the data with same kind of attributes and characteristics. You can also think of this is some kind of database schema. Splunk had already pre-built a CIM app which comes with many excellent data model and field name.

For instance, SOC usually ingest a lot of different raw log into SIEM system including proxy, firewall and IDS. These logs do not share the same attributes, but proxy log always share similar attributes. Therefore, Splunk defined a data model named Web and its attributes (url, http_user_agent and http_referrer). What you have to do is simply assign tag=web and tag=proxy to those proxy logs. Then, Splunk will automatically recognize those log as Web Data Model.

In my experience, the design of Data Model always contribute to the success of SOC. Below list out some point worth to note:

• Data Model Acceleration
  • An excellent feature to store summary data (NOT raw log) with reduced disk space.
  • Not ALL Data Model are accelerated by default
  • Useful for long term reporting and dashboard as the speed is excellent
• Data Model tag whitelist
  • Each Data Model has its own tag whitelist. You must remember this when modifying existing data model. Do not directly edit configuration file. Please refer to link here.
• Use same field name
  • Align to the field name used by Splunk when developing new app or new log type. It an significantly reduce the learning curve of SOC analyst.
  • ALL the field name within Data Model are small capital
• Reuse of existing use-cases/correlation rules
  • If we have enabled “brute-force attack” using Authentication Data Model, detection will automatically take place whenever a new device is onboarded with correct data model and field name defined.

Threat – URL Matches – Threat Gen

Now, we will take a look of “Threat – URL Matches – Threat Gen” search. Basically, this search will only search threat intel within Web Data Model. So, do not expect it will look for elsewhere.

| `tstats` values(sourcetype) as sourcetype,values(Web.src),values(Web.dest) from datamodel=Web.Web by Web.http_referrer | eval url='Web.http_referrer' | eval threat_match_field="http_referrer" | `tstats` append=true values(sourcetype) as sourcetype,values(Web.src),values(Web.dest) from datamodel=Web.Web by Web.url | eval url=if(isnull(url),'Web.url',url) | eval threat_match_field=if(isnull(threat_match_field),"url",threat_match_field) | stats values(sourcetype) as sourcetype,values(Web.src) as src,values(Web.dest) as dest by url,threat_match_field | extract domain_from_url | `threatintel_url_lookup(url)` | `threatintel_domain_lookup(url_domain)` | search threat_collection_key=* | `mvtruncate(src)` | `mvtruncate(dest)` | `zipexpand_threat_matches`

There is another search “Threat – Source And Destination Matches – Threat Gen” which look for 3 data model. If you read through this article, I believe you can locate it yourself. Leave us a message if you encounter any issues.

Threat Activity Detected

Finally, we comes to the last steps of Splunk Threat intel framework. There is not much to discuss here. The correlation rule “Threat Activity Detected” should simply works after enabled it. The “Incident Review” page will show the “Threat Source ID”, which indicate where the threat intel comes from. An SOC analyst should first check this field to determine their next step or maybe some junior analyst add a wrong records.

Additional design consideration

In this final section, I want to cover some additional consideration when designing SIEM correlation rules. It is more general and not limited to “Threat Activity Detected” correlation rule.

Generally speaking, each correlation rule is configured to search for events occurred within a certain period (e.g. from 1 hour ago to now). There is no issues if our world is perfect. In the real world, we do experience many issues such as indexer performance issues and cloud log delay issues. Of course, we cannot consider every situation, but at least if we know the logs from AWS is having an average of 20 minutes delay. Then, we have to consider about it. You can simply start to examine the reasonable earliest and latest value by the following query.

index=*
|  eval indextime=_indextime
|  eval timediff=indextime-_time
|  stats count, max(timediff), avg(timediff) by sourcetype, source, host

Thanks very much to read this long long articles. Stay safe!