[azure,o365,m365_defender] ECS mapping improvements #14085

chrisberkhout · 2025-05-30T14:00:14Z

Proposed commit message

[azure,o365,m365_defender] ECS mapping improvements

Various suggested ECS field mapping updates to improve correlation of
ingested events with other security telemetry, for the integrations
azure, o365 and m365_defender.

Discussion

Can be read commit-by-commit.

Suggested mappings done:

Suggestion: azure.signinlogs.properties.user_principal_name -> user.email
Done, whenever the value contains @.
Suggestion: azure.signinlogs.properties.service_principal_name -> service.name
Done, whenever it's populated.
Suggestion: azure.identityprotection.properties.user_principal_name -> user.email
Suggestion: azure.identityprotection.properties.user_display_name -> user.full_name
Suggestion: azure.identityprotection.properties.user_id -> user.id
All done. Matched to the handling in the signinlogs data stream.
Suggestion: o365.audit.OriginatingDomain -> url.domain
Suggestion: o365.audit.Application -> process.name
Suggestion: o365.audit.Sha1 -> file.hash.sha1
Suggestion: o365.audit.Sha256 -> file.hash.sha256
Suggestion: o365.audit.FileExtension -> file.extension
Suggestion: o365.audit.Parameters.From -> email.from.address
All done. Field defintion and mapping logic added.
Suggestion: o365.audit.DeviceName -> host.name
Done. DeviceName is now the first source for host.name and two other sources will be tried if it’s empty.
Suggestion: o365.audit.TargetFilePath -> file.path
Done. Also FilePath.
Suggestion: o365.audit.FileSize -> file.size
Done. Field definition for FileSize added. New mapping logic tries FileSizeBytes first and falls back to FileSize.
Suggestion: o365.audit.Parameters.ForwardAsAttachmentTo -> email.to.address
Suggestion: o365.audit.Parameters.ForwardTo -> email.to.address
Suggestion: o365.audit.Parameters.RedirectTo -> email.to.address
All done. Mapping logic added.

Suggested mappings not done:

Suggestion: o365.audit.UserId -> user.email
No change. UserId was already mapped to user.id here, and set in user.email if the value has @ in it, here. The user.email field can get other values if UserId is not set.
Suggestion: m365_defender.incident.alerts.evidence.user_account.azure_ad_user_id -> user.id
Suggestion: m365_defender.incident.alerts.evidence.user_account.user_sid -> user.id
Suggestion: m365_defender.incident.alerts.evidence.user_account.user_principal_name -> user.email
Not done. These aren't the user the event relates to. The incident can have related alerts, each with several pieces of related evidence, each with a related user account. These IDs/names are already copied to related.user, which should facilitate matching in searches and rules. Duplicating this information in user.* seems inappropriate.
Suggestion: m365_defender.alert.evidence.p1_sender.display_name -> user.full_name
Suggestion: m365_defender.alert.evidence.p2_sender.display_name -> user.full_name
Not done. As above, these are already added to related.user. This is the case in both the alert and incident data streams.
Suggestion: m365_defender.incident.alerts.evidence.display_name -> user.full_name
Not done. This is not always a user name. For example, it's a group name in this case. This could be revisited if we have the data to know exactly when it's a user.

Suggested mappings done with adjustments:

Suggestion: m365_defender.incident.alerts.evidence.user_account.display_name -> user.full_name
Added this to related.user, to match the handling of azure_ad_user_id, user_sid, and user_principal_name.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Closes [azure,o365,m365_defender] ECS mapping improvements #13989

elasticmachine · 2025-05-30T14:29:49Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6

LGTM if you are happy with the queries?

packages/azure/data_stream/identity_protection/elasticsearch/ingest_pipeline/default.yml

packages/m365_defender/data_stream/incident/elasticsearch/ingest_pipeline/default.yml

efd6 · 2025-06-02T06:31:59Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+  - gsub:
+      field: file.size
+      pattern: ','
+      replacement: ''
+      ignore_missing: true


Are they always localized to non-European thousands separators?

Not sure. One value was provided with the suggestion; "3,818,355".

In the MS documentation* I see:

In existing pipeline test input we have this (it's a different place in the document):

{ "ExchangeMetaData": { "FileSize": 13405, ... }, ... }

I think more resilient logic would be:

convert to string (to add)

remove non-digits (this gsub modified)

convert to long (existing)

Also changed the new field definition to keyword.

elastic-vault-github-plugin-prod · 2025-06-02T08:45:12Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6 · 2025-06-03T21:33:14Z

packages/azure/data_stream/identity_protection/elasticsearch/ingest_pipeline/default.yml

+  - set:
+     field: user.email
+     copy_from: azure.identityprotection.properties.user_principal_name
+     if: "(ctx.azure?.identityprotection?.properties?.user_principal_name ?: '') =~ /^[^@]+@[^@]+$/"


I'm disinclined to do more than checking for an @; "I am the @ man"@gmail.com is a valid email address per the RFC.

Oh, good to know.

Switched back to the simpler check.

…to related.user.

…ace.

…String.

efd6

Still LGTM

muthu-mps

Let a comment to modify the changelog description.
Reviewed Azure signinlogs, LGTM!

packages/azure/changelog.yml

ishleenk17 · 2025-06-05T05:49:45Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+      field: file.size
+      copy_from: o365audit.FileSize
+      ignore_empty_value: true
+      override: false


Can only one of filesize and filesizebytes can exist ?

The current API documentation includes FileSize, with the description "Size for the file in bytes", so I think these fields have the same meaning.

The FileSizeBytes mapping was added in a community PR, based on production data. I don't see the field in the documentation. It may be that the field name changed or that it only exists in certain cases.

We have a user request to parse FileSize, so I want to prefer that but I to fall back to the original FileSizeBytes parsing in case it does appear with a different version or configuration, or in older data.

ishleenk17 · 2025-06-05T05:52:30Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+  - convert:
+      tag: convert_file_size_to_long
+      field: file.size
+      type: long


Converting to string before and then to long ?
Could you explain more ?

The idea was to have logic that works for values from two sources:

FileSize: documented as a string, provided sample value of "3,818,355"

FileSizeBytes: undocumented, mapped as a long with no conversion

But on second thought, we can remove the string conversion and limit the removal of non-digit characters to values that are already strings, so I've done that.

Note: there is an issue (elastic/beats#43659, elastic/elasticsearch#128160) with converting numbers of a certain size to long, but both versions have that and it's being fixed in ES.

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

elasticmachine · 2025-06-05T10:27:49Z

💚 Build Succeeded

Buildkite Build
Commit: 5819ab1

History

💔 Build #26921 failed 223af98
💚 Build #26845 succeeded 0624b35
💚 Build #26792 succeeded 3b51ff3
💔 Build #26790 failed 645c0d4
💔 Build #26716 failed e5a7d70

cc @chrisberkhout

elastic-sonarqube · 2025-06-05T10:27:58Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
95.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

zmoog

LGTM!

elastic-vault-github-plugin-prod · 2025-06-07T08:03:04Z

Package azure - 1.26.0 containing this change is available at https://epr.elastic.co/package/azure/1.26.0/

elastic-vault-github-plugin-prod · 2025-06-07T08:03:04Z

Package m365_defender - 3.9.0 containing this change is available at https://epr.elastic.co/package/m365_defender/3.9.0/

elastic-vault-github-plugin-prod · 2025-06-07T08:03:05Z

Package o365 - 2.18.0 containing this change is available at https://epr.elastic.co/package/o365/2.18.0/

chrisberkhout self-assigned this May 30, 2025

chrisberkhout added enhancement New feature or request Integration:azure Azure Logs Integration:o365 Microsoft Office 365 Integration:m365_defender Microsoft Defender XDR Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels May 30, 2025

chrisberkhout force-pushed the ecs-suggested-mapping-improvments branch 2 times, most recently from 920c94b to 469ab90 Compare May 30, 2025 14:29

chrisberkhout marked this pull request as ready for review May 30, 2025 14:29

chrisberkhout requested review from a team as code owners May 30, 2025 14:29

andrewkroh added the Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] label May 30, 2025

efd6 approved these changes Jun 2, 2025

View reviewed changes

chrisberkhout force-pushed the ecs-suggested-mapping-improvments branch from fae9079 to e5a7d70 Compare June 2, 2025 08:19

chrisberkhout requested a review from efd6 June 2, 2025 08:20

chrisberkhout force-pushed the ecs-suggested-mapping-improvments branch 2 times, most recently from 2a59c2f to 3b51ff3 Compare June 3, 2025 10:08

efd6 reviewed Jun 3, 2025

View reviewed changes

chrisberkhout added 9 commits June 4, 2025 09:29

Adjust azure fields.

ebc36e1

Rebuild azure docs covering identity_protection and signinlogs.

ea80a78

Define additional fields for o365. Rebuild README.

b5cc30f

Adjust various o365 mappings.

1ac238e

Add m365_defender.incident.alerts.evidence.user_account.display_name …

5f589ca

…to related.user.

Version bumps, changelog entries.

afcd791

Update expected pipeline test output for m365_defender.

e3ecbe9

Remove redundant validation of '@' in user principal name. Fix whitep…

0f91c63

…ace.

More resilient processing of FileSize/FileSizeBytes.

2aa1118

chrisberkhout added 3 commits June 4, 2025 09:29

Make new FileSize field keyword to match MS documentated type of Edm.…

c2ed810

…String.

Check format for user.email values.

e4f35c2

Back to the simpler email address check.

0624b35

chrisberkhout force-pushed the ecs-suggested-mapping-improvments branch from 3b51ff3 to 0624b35 Compare June 4, 2025 07:32

chrisberkhout enabled auto-merge (squash) June 4, 2025 07:35

chrisberkhout disabled auto-merge June 4, 2025 07:35

efd6 approved these changes Jun 4, 2025

View reviewed changes

muthu-mps approved these changes Jun 5, 2025

View reviewed changes

packages/azure/changelog.yml Outdated Show resolved Hide resolved

ishleenk17 reviewed Jun 5, 2025

View reviewed changes

chrisberkhout and others added 2 commits June 5, 2025 11:24

Update packages/azure/changelog.yml

223af98

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

Simplify file.size conversion.

5819ab1

chrisberkhout requested a review from ishleenk17 June 5, 2025 11:54

zmoog approved these changes Jun 6, 2025

View reviewed changes

chrisberkhout merged commit c5b49d5 into elastic:main Jun 7, 2025
8 checks passed

zmoog mentioned this pull request Jun 10, 2025

all: standardize user name and user email fields across integrations #14137

Merged

5 tasks

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025

[azure,o365,m365_defender] ECS mapping improvements #14085

[azure,o365,m365_defender] ECS mapping improvements #14085

Uh oh!

Conversation

chrisberkhout commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Discussion

Suggested mappings done:

Suggested mappings not done:

Suggested mappings done with adjustments:

Checklist

Related issues

Uh oh!

elasticmachine commented May 30, 2025

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

efd6 Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

chrisberkhout Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

chrisberkhout Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod bot commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

efd6 Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisberkhout Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

muthu-mps left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ishleenk17 Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

chrisberkhout Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ishleenk17 Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

chrisberkhout Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jun 5, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Jun 5, 2025

Quality Gate passed

Uh oh!

zmoog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Jun 7, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jun 7, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jun 7, 2025

Uh oh!

Uh oh!

chrisberkhout commented May 30, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Jun 2, 2025 •

edited

Loading

efd6 Jun 3, 2025 •

edited

Loading

chrisberkhout Jun 5, 2025 •

edited

Loading