[windows] Defender - Fix pipeline for Path and add missing fields.

[nicpenning]

It was reported in the community that the File Path will sometimes contain a process pid which is not currently processor correctly (see screenshots). This PR will resolve this issue and also add more fields that were missed in the last update.

I also added 2 more test documents to add the other potential data that would arrive.

Convo started here: https://elasticstack.slack.com/archives/C018PDGK6JU/p1729764392118379?thread_ts=1729593410.515049&cid=C018PDGK6JU

• Bug

This will improve the integration further.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

Screenshots

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

[nicpenning]

Came across another wrench in the pipeline. Sometimes the Path may contain multiple "Files".

Microsoft Defender Antivirus has detected malware or other potentially unwanted software.
 For more information please see the following:
https://go.microsoft.com/fwlink/?linkid=37020&name=TrojanSpy:MSIL/AgentTesla!MSR&threatid=2147753554&enterprise=1
 	Name: TrojanSpy:MSIL/AgentTesla!MSR
 	ID: 2147753554
 	Severity: Severe
 	Category: Trojan Monitoring Software
 	Path: file:_D:\autorun.inf\autorun.inf                                                                                                                                           .exe; file:_D:\autorun.inf\Protection for Autorun\Protection for Autorun                                                                                                                                .exe; file:_D:\ \                                                                                                                                                      .exe
 	Detection Origin: Local machine
 	Detection Type: Concrete
 	Detection Source: Real-Time Protection
 	User: YAMS\WIN1337
 	Process Name: C:\Windows\explorer.exe
 	Security intelligence Version: AV: 1.419.578.0, AS: 1.419.578.0, NIS: 1.419.578.0
 	Engine Version: AM: 1.1.24080.9, NIS: 1.1.24080.9

Not sure what the deal with the parsing is, but this is the raw JSON captured in Kibana. Maybe that is the actual file name. More research needed:

        "Path": "file:_D:\\autorun.inf\\autorun.inf                                                                                                                                           .exe; file:_D:\\autorun.inf\\Protection for Autorun\\Protection for Autorun                                                                                                                                .exe; file:_D:\\ \\                                                                                                                                                      .exe",

[nicpenning]

So this pipeline will still populate the first file.path found but won't include the others found.

How should this be handled if related.file.name does not seem to exist and file.name is supposed to be a keyword?

My thought is that I could have a custom field called windows_defender.files_detected and make it an array type.

Thoughts?

Otherwise, good to review and test.

[leehinman]

So this pipeline will still populate the first file.path found but won't include the others found.

How should this be handled if related.file.name does not seem to exist and file.name is supposed to be a keyword?

My thought is that I could have a custom field called windows_defender.files_detected and make it an array type.

Thoughts?

Otherwise, good to review and test.

related.files might be a good approach. Similar to ECS related.ip or related.user. https://www.elastic.co/guide/en/ecs/current/ecs-related.html

[leehinman]

LGTM so far. Will review again when an approach is decided on the multiple files issue.

[nicpenning]

So this pipeline will still populate the first file.path found but won't include the others found.
How should this be handled if related.file.name does not seem to exist and file.name is supposed to be a keyword?
My thought is that I could have a custom field called windows_defender.files_detected and make it an array type.
Thoughts?
Otherwise, good to review and test.

related.files might be a good approach. Similar to ECS related.ip or related.user. https://www.elastic.co/guide/en/ecs/current/ecs-related.html

Would related.file.name actually work? I'm concerned this doesn't fit within the bounds of ECS since it's not called out in the reference. I would do that in a heart beat if related can be applied to any logical ECS field.

Thoughts?

[leehinman]

So this pipeline will still populate the first file.path found but won't include the others found.
How should this be handled if related.file.name does not seem to exist and file.name is supposed to be a keyword?
My thought is that I could have a custom field called windows_defender.files_detected and make it an array type.
Thoughts?
Otherwise, good to review and test.

related.files might be a good approach. Similar to ECS related.ip or related.user. https://www.elastic.co/guide/en/ecs/current/ecs-related.html

Would related.file.name actually work? I'm concerned this doesn't fit within the bounds of ECS since it's not called out in the reference. I would do that in a heart beat if related can be applied to any logical ECS field.

Thoughts?

I was originally thinking a new field name, outside of ECS, but to use the related fields as inspiration. (Sorry I wasn't clear on that)

But maybe this is a useful enough construct that an rfc to add it would be a good idea?

[nicpenning]

Understood!

I will create a new field. Should I stick with winlog.*? Or should we think about moving to something more specific for the windows defender fields like I noted before: windows_defender.*

I could then migrate any others that were the generic winlog.eventdata and map to this new subfield.

Also, that RFC process looks pretty daunting at this time so I won't pursue that but I do think it is a good idea to include related.file.*

[nicpenning]

Alright, I had to sync other changes to the integration and then use a new windows_defender field. Please review!

[nicpenning]

Please test.

[jamiehynds]

@leehinman are you ok to continue with the review here, or prefer if @marc-gr takes over as it's security releated?

[leehinman]

LGTM

[leehinman]

@leehinman are you ok to continue with the review here, or prefer if @marc-gr takes over as it's security releated?

LGTM and approved but if @marc-gr could do a double check that would be good. I'm rusty when it comes to integrations :-)

[nicpenning]

Hello, could we get some eyes on this again? Getting a bit stale :)

[marc-gr]

/test

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: c80e649

[elastic-vault-github-plugin-prod]

Package windows - 2.3.1 containing this change is available at https://epr.elastic.co/package/windows/2.3.1/

[nicpenning]

Thank you!