azure: add related.entity field to auditlogs and signinlogs default ingest pipeline

[orouz]

Proposed commit message

adds processors to auditlogs and signinlogs ingest pipelines that appends all entity identifiers from the log event to related.entity in order to facilitate pivoting around a piece of data

Context

this PR is part of the cloud security CDR epic. it populates related.entity which is an upcoming ECS field

benchmark (100 runs)

1. auditlogs

Metric
Description	Main	append + painless script processor
Commit	63854f0	01e0f3c
Average EPS	16409.48978	15400.45392
Stddev	1273.233006	1197.630202
Min	8196.721311	10416.66667
Max	17857.14286	16949.15254
EPS Change to baseline		-6.15%

2. signinlogs

Metric
Description	Main	append processor
Commit	63854f0	de9827c
Average EPS	9641.707196	9561.806718
Stddev	701.7646604	861.931866
Min	7142.857143	5405.405405
Max	11363.63636	10989.01099
EPS Change to baseline		-0.83%

Related issues

• closes https://github.com/elastic/security-team/issues/10391

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package azure 👍(4) 💚(4) 💔(3)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
signinlogs	2392.34	1937.98	-454.36 (-18.99%)	💔
eventhub	500000	333333.33	-166666.67 (-33.33%)	💔
firewall_logs	1869.16	1406.47	-462.69 (-24.75%)	💔

To see the full report comment with /test benchmark fullreport

[orouz]

/test benchmark fullreport

[efd6]

@orouz Please revise the proposed commit message to be a stand-alone text that explains what is being done and why. It should preferably not refer to internal issues since it will be read by people who may not have access to those documents. Also note that git commit messages are not markdown, so do not use markdown features.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
81.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: d6c726e

History

• 💚 Build #17376 succeeded 06cb8f6
• 💚 Build #17375 succeeded 06cb8f6
• 💔 Build #17372 failed 01e0f3c7a859337864ac32e584ad5a54c04354a5
• 💔 Build #16856 failed 62557f7cf362e3fb0c83ec3486c6c8a1034ebfe6
• 💔 Build #16852 failed 5c5457aef3de900ba731c0a0c9816b5e1c675138

cc @orouz

[elastic-vault-github-plugin-prod]

Package azure - 1.18.0 containing this change is available at https://epr.elastic.co/search?package=azure

[nicpenning]

related.entity doesn't appear to be a field that will be in compliance with ECS.

https://www.elastic.co/guide/en/ecs/current/ecs-related.html

I have an integration going through the works right now that would like to use something like related.file.name but was directed to creating an RFC for adopting that field.

Will this cause issues down the road or those that use the Data Quality dashboard to reveal actual ECS fields? Is there already a precedent set here that any field can live beneath related.? Thoughts?

[orouz]

related.entity doesn't appear to be a field that will be in compliance with ECS.

it's currently being worked on at elastic/ecs#2360

Will this cause issues down the road or those that use the Data Quality dashboard to reveal actual ECS fields? Is there already a precedent set here that any field can live beneath related.? Thoughts?

@tinnytintin10 can comment more on that.