[windows] Windows Defender Antivirus Data Stream - Initial Add

[nicpenning]

• Enhancement

Add Windows Defender data stream - more details to come.

Proposed commit message

See title.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Relates [Windows] Simplifying Windows Events packages #4564
• Relates Windows Defender Antivirus #1729

Screenshots

[nicpenning]

Have about 28 of these: /integrations/build/packages/windows-1.46.0.zip/data_stream/windows_defender/fields/winlog.yml" is invalid: field 0.fields.4.fields.111.name: Does not match pattern '^[-_/@A-Za-z0-9]+(.[-_/@A-Za-z0-9]+)*$'

Since there are quite a few fields with spaces. Pipelines will need to be created to handle these since you cannot have spaces in field names for an integration, even though you can have spaces in fields names in Elasticsearch in general.

[elasticmachine]

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

[nicpenning]

If anyone knows how to rename a field in a pipeline with spaces I would gladly hear it.

I would like to do something like this, but unsure if it will work.

- rename:
  field: "winlog.eventdata.AS security intelligence creation time"
  target_field: "winlog.eventdata.AS_security_intelligence_creation_time"
  ignore_failure: true
  ignore_missing: true
  if: "winlog.eventdata.AS security intelligence creation time" != null

Please advise, as this is nearly ready for testing - The spaces in field names is the remaining cleanup item before tests can begin.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[nicpenning]

Replaced spaces with _ for field names, and looks good.

Ready to test!

[belimawr]

LGMT.

Regarding remove the spaces from field names, you can also use the script processor that allows you to freely manipulate the data and have a generic solution instead of hardcoding all field names.

Filebeat also has got a script processor in case you want that done by Filebeat instead of ES.

[nicpenning]

@leehinman , @belimawr

Maybe something like this? Works, but unsure if efficient.

Script processor for ingest pipeline. Doing this at the agent level would be more efficient, but unsure how to bake this into the integration on the filebeat side (the context of how this code operates).

if (ctx.winlog?.event_data != null) {
  Map eventData = ctx.winlog.event_data;
  Map newEventData = new HashMap();
  for (entry in eventData.entrySet()) {
    String newKey = entry.getKey().replace(" ", "_");
    newEventData.put(newKey, entry.getValue());
  }
  ctx.winlog.event_data = newEventData;
}

With this conditional as well

[nicpenning]

Moved processing to client side, please review!

[belimawr]

LGTM.

@nicpenning is there an easy way to test this PR? How can I generate the necessary data to test it?

I really would like to test this JS script, just to be on the safe side.

[nicpenning]

I tested locally on my device and it seemed to work.

I do dev on Windows so I exposed my containerized stack to my localhost and installed an agent on my windows host and pointed to fleet:8220 using 127.0.0.1:8220 in the fleet settings (new fleet server) and elasticsearch:9200 using 127.0.0.1:9200 (new elasticsearch output) in the output options

[nicpenning]

So I did a manual test and @leehinman's code for the preserve original event did not work. I reverted back to what the other Windows datastreams had and it is now functioning. Please advise!

[leehinman]

So I did a manual test and @leehinman's code for the preserve original event did not work. I reverted back to what the other Windows datastreams had and it is now functioning. Please advise!

What was the test, and how did it fail? I did check what you had merged and at least at the policy level it looked like the tags were being added correctly.

[nicpenning]

I deployed the agent on my Windows workstation manually with those settings and the event.original was not being added to the event.

[leehinman]

I deployed the agent on my Windows workstation manually with those settings and the event.original was not being added to the event.

When you added the suggestion, you removed this section:

{{#if preserve_original_event}}
include_xml: true
{{/if}}

without that, the winlog input won't give you the original event. The tags part is so that the default ingest pipeline doesn't delete it.

[nicpenning]

Whoops! I will give it go as soon as I can! Sorry about that!

[nicpenning]

Is this what I need for the entirety of the file then? Looks like my tests had this:

name: Microsoft-Windows-Windows Defender/Operational
condition: ${host.platform} == 'windows'
{{#if event_id}}
event_id: {{event_id}}
{{/if}}
{{#if ignore_older}}
ignore_older: {{ignore_older}}
{{/if}}
{{#if language}}
language: {{language}}
{{/if}}

{{#if tags.length}}
tags:
{{#each tags as |tag|}}
  - {{tag}}
{{/each}}
{{#if preserve_original_event}}
  - preserve_original_event
{{/if}}
{{else}}
{{#if preserve_original_event}}
tags:
  - preserve_original_event
{{/if}}
{{/if}}
{{#if preserve_original_event}}
include_xml: true
{{/if}}
processors:
- translate_sid:
    field: winlog.event_data.MemberSid
    account_name_target: winlog.event_data._MemberUserName
    domain_target: winlog.event_data._MemberDomain
    account_type_target: winlog.event_data._MemberAccountType
    ignore_missing: true
    ignore_failure: true
- script:
    lang: javascript
    id: rename_fields_in_winlog_event_data_that_have_spaces
    source: >
      function process(event) {
        var eventData = event.Get("winlog.event_data");
        if (eventData !== null) {
          var newEventData = {};
          for (var key in eventData) {
            if (eventData.hasOwnProperty(key)) {
              var newKey = key.replace(/ /g, "_");
              newEventData[newKey] = eventData[key];
            }
          }
          event.Put("winlog.event_data", newEventData);
        }
      }
{{#if processors.length}}
{{processors}}
{{/if}}
{{custom}}

[leehinman]

Is this what I need for the entirety of the file then? Looks like my tests had this:

Sorry, I meant when you tests the code using unless that removed the ambiguous else statements.
Here is the snippet before the processors part of the file

name: Microsoft-Windows-Windows Defender/Operational
condition: ${host.platform} == 'windows'
{{#if event_id}}
event_id: {{event_id}}
{{/if}}
{{#if ignore_older}}
ignore_older: {{ignore_older}}
{{/if}}
{{#if language}}
language: {{language}}
{{/if}}

{{#if preserve_original_event}}
include_xml: true
{{/if}}

# if we have tags defined add the tags object
{{#if tags.length}}
tags:
{{/if}}

# if tags aren't defined but preserver original is, add tags object
{{#unless tags.length}}
{{#if preserve_original_event}}
tags:
{{/if}}
{{/unless}}

# if tags are defined, add them
{{#if tags.length}}
{{#each tags as |tag|}}
  - {{tag}}
{{/each}}
{{/if}}

# if preserve original event is defined add that tag
{{#if preserve_original_event}}
  - preserve_original_event
{{/if}}

[nicpenning]

Re-added code. Will test later this evening. If you have a chance to give it a go again and validate the code looks good that would be great.

[nicpenning]

Local tests look good. I can't get my manual tests to work. I am good to ship it as is with the built in tests passing.

[nicpenning]

@leehinman - Any other updates needed here?

Keep in mind that I did mark this data stream as Beta. Can take another run at it with any issues/fixes before dropping that flag on a future PR.

[leehinman]

@leehinman - Any other updates needed here?

Keep in mind that I did mark this data stream as Beta. Can take another run at it with any issues/fixes before dropping that flag on a future PR.

Not from me, I "approved". I think you need someone from "sec-windows-platform" to review since they are codeowners.

[nicpenning]

@efd6 - Is this your team?

[pierrehilbert]

Hey @nfritts, could we have someone from your team to review here?

[marc-gr]

/test

[marc-gr]

/test

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f7cc903

History

• 💔 Build #13589 failed 8cbfe17

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
95.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Package windows - 1.46.0 containing this change is available at https://epr.elastic.co/search?package=windows