abnormal_security: add support for Not Analyzed Messages data stream #13483

chemamartinez · 2025-04-09T07:37:23Z

Proposed commit message

Add support for AI Security Mailbox Not Analyzed data stream. API reference can be found here.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Closes [abnormal_security] Add Support for Not Analyzed Messages in Abuse Mailbox #13000

Screenshots

elasticmachine · 2025-04-09T07:38:33Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

chemamartinez · 2025-04-09T09:57:49Z

/test

efd6 · 2025-04-10T03:09:35Z

packages/abnormal_security/_dev/deploy/docker/files/config.yml

+          Content-Type:
+            - 'application/json'
+        body: |-
+          {


Suggested change

{

{{ minify_json `

{

efd6 · 2025-04-10T03:09:46Z

packages/abnormal_security/_dev/deploy/docker/files/config.yml

+                      "not_analyzed_reason": "PHISHING_SIMULATION"
+                  }
+              ]
+          }


Suggested change

}

}

`}}

efd6 · 2025-04-10T03:10:53Z

...i_security_mailbox_not_analyzed/_dev/test/pipeline/test-ai-security-mailbox-not-analyzed.log

@@ -0,0 +1,3 @@
+{"abx_message_id":-7361381340273523750,"recipient":{"email":"phishing@test.com","name":"Phishing Test"},"reported_datetime":"2025-03-06T17:27:15Z","reporter":{"email":"reporter@test.com","name":"Reporter Test"},"subject":"Re: Subject","not_analyzed_reason":"ROUTED_SUBMISSION"}
+{"abx_message_id":240750237502375023,"recipient":{"email":"phishing@test.com","name":"Phishing Test"},"reported_datetime":"2025-03-04T18:50:27Z","reporter":{"email":"test@example.com","name":"Test Example"},"subject":"Fw: Forwarded email","not_analyzed_reason":"INVALID_SUBMISSION"}
+{"abx_message_id":-1234567891234567891,"recipient":{"email":"phishing@test.com","name":"Phishing Test"},"reported_datetime":"2025-03-04T17:03:55Z","reporter":{"email":"info@test.com","name":"Info Test"},"subject":"Fwd: Forwarded email","not_analyzed_reason":"PHISHING_SIMULATION"}


Add final new line.

efd6 · 2025-04-10T03:12:34Z

packages/abnormal_security/_dev/deploy/docker/files/config.yml

@@ -101,6 +101,63 @@ rules:
              "attackType": "Attack Type: Spam"
          }
          `}}
+  - path: /v1/abuse_mailbox/not_analyzed


Can I confirm that the API is non-paginating?

Looks like it can take start and end datetimes

Yes, no pagination parameters are accepted for this endpoint. Here's the reference.

efd6 · 2025-04-10T03:13:14Z

...ages/abnormal_security/data_stream/ai_security_mailbox_not_analyzed/agent/stream/cel.yml.hbs

+        "Authorization": ["Bearer " + string(state.access_token)],
+      }
+    }).do_request().as(resp, resp.StatusCode == 200 ?
+        bytes(resp.Body).decode_json().as(body, has(body.results) && size(body.results) > 0 ?


Suggested change

bytes(resp.Body).decode_json().as(body, has(body.results) && size(body.results) > 0 ?

resp.Body.decode_json().as(body, has(body.results) && size(body.results) > 0 ?

efd6 · 2025-04-10T03:14:01Z

...ages/abnormal_security/data_stream/ai_security_mailbox_not_analyzed/agent/stream/cel.yml.hbs

+            "error": {
+              "code": string(resp.StatusCode),
+              "id": string(resp.Status),
+              "message": "GET:"+(


Suggested change

"message": "GET:"+(

"message": "GET "+state.url.trim_right("/") + "/v1/abuse_mailbox/not_analyzed:"+(

efd6 · 2025-04-10T03:15:16Z

...ages/abnormal_security/data_stream/ai_security_mailbox_not_analyzed/agent/stream/cel.yml.hbs

+        "start": [string(state.start_time)],
+        "end": [string(state.end_time)],


Suggested change

"start": [string(state.start_time)],

"end": [string(state.end_time)],

"start": [state.start_time],

"end": [state.end_time],

These have just been formatted to strings, in the preamble block, so the conversion is not necessary.

efd6 · 2025-04-10T03:18:32Z

...normal_security/kibana/dashboard/abnormal_security-37ed5d19-c753-43a0-b0a2-f8e6437ddfe5.json

@@ -116,7 +116,7 @@
                        "id": "",
                        "params": {
                            "fontSize": 12,
-                            "markdown": "**Navigation**\n\n**Abnormal Security**\n\n- [AI Security Mailbox Overview](#/dashboard/abnormal_security-a4364503-ada3-4fe6-a054-d152accf207c)\n- [**Audit Overview (This Page)**](#/dashboard/abnormal_security-37ed5d19-c753-43a0-b0a2-f8e6437ddfe5)\n- [Case Overview](#/dashboard/abnormal_security-f6562262-e429-470d-af45-4c80afdcf664)\n- [Threat Overview](#/dashboard/abnormal_security-6a8e53ac-7759-4564-bcd6-03c6a9792eac)\n- [Vendor Case Overview](#/dashboard/abnormal_security-7997c0a4-da55-4090-b24f-586dbd19aff4)\n\n**Overview**\n\nThis dashboard displays key statistics and visualizations based on Audit logs from the Abnormal Security integration. It includes the top 10 users and source IPs, a breakdown of events by category, action and tenant name, as well as essential details about the Audit data.\n\n[**Integrations Page**](/app/integrations/detail/abnormal_security/overview)",
+                            "markdown": "**Navigation**\n\n**Abnormal Security**\n\n- [AI Security Mailbox Overview](#/dashboard/abnormal_security-a4364503-ada3-4fe6-a054-d152accf207c)\n- [AI Security Mailbox Not Analyzed Overview](#/dashboard/abnormal_security-a0e8aab9-b870-4903-a966-7195fd6cee9c)\n- [**Audit Overview (This Page)**](#/dashboard/abnormal_security-37ed5d19-c753-43a0-b0a2-f8e6437ddfe5)\n- [Case Overview](#/dashboard/abnormal_security-f6562262-e429-470d-af45-4c80afdcf664)\n- [Threat Overview](#/dashboard/abnormal_security-6a8e53ac-7759-4564-bcd6-03c6a9792eac)\n- [Vendor Case Overview](#/dashboard/abnormal_security-7997c0a4-da55-4090-b24f-586dbd19aff4)\n\n**Overview**\n\nThis dashboard displays key statistics and visualizations based on Audit logs from the Abnormal Security integration. It includes the top 10 users and source IPs, a breakdown of events by category, action and tenant name, as well as essential details about the Audit data.\n\n[**Integrations Page**](/app/integrations/detail/abnormal_security/overview)",


Suggested change

"markdown": "**Navigation**\n\n**Abnormal Security**\n\n- [AI Security Mailbox Overview](#/dashboard/abnormal_security-a4364503-ada3-4fe6-a054-d152accf207c)\n- [AI Security Mailbox Not Analyzed Overview](#/dashboard/abnormal_security-a0e8aab9-b870-4903-a966-7195fd6cee9c)\n- [**Audit Overview (This Page)**](#/dashboard/abnormal_security-37ed5d19-c753-43a0-b0a2-f8e6437ddfe5)\n- [Case Overview](#/dashboard/abnormal_security-f6562262-e429-470d-af45-4c80afdcf664)\n- [Threat Overview](#/dashboard/abnormal_security-6a8e53ac-7759-4564-bcd6-03c6a9792eac)\n- [Vendor Case Overview](#/dashboard/abnormal_security-7997c0a4-da55-4090-b24f-586dbd19aff4)\n\n**Overview**\n\nThis dashboard displays key statistics and visualizations based on Audit logs from the Abnormal Security integration. It includes the top 10 users and source IPs, a breakdown of events by category, action and tenant name, as well as essential details about the Audit data.\n\n[**Integrations Page**](/app/integrations/detail/abnormal_security/overview)",

"markdown": "**Navigation**\n\n**Abnormal Security**\n\n- [AI Security Mailbox Overview](#/dashboard/abnormal_security-a4364503-ada3-4fe6-a054-d152accf207c)\n- [AI Security Mailbox Not Analyzed Overview](#/dashboard/abnormal_security-a0e8aab9-b870-4903-a966-7195fd6cee9c)\n- **Audit Overview**\n- [Case Overview](#/dashboard/abnormal_security-f6562262-e429-470d-af45-4c80afdcf664)\n- [Threat Overview](#/dashboard/abnormal_security-6a8e53ac-7759-4564-bcd6-03c6a9792eac)\n- [Vendor Case Overview](#/dashboard/abnormal_security-7997c0a4-da55-4090-b24f-586dbd19aff4)\n\n**Overview**\n\nThis dashboard displays key statistics and visualizations based on Audit logs from the Abnormal Security integration. It includes the top 10 users and source IPs, a breakdown of events by category, action and tenant name, as well as essential details about the Audit data.\n\n[**Integrations Page**](/app/integrations/detail/abnormal_security/overview)",

(similar below)

What is the motivation for this? To remove the (This page) message next to the dashboard title for everyone?

Would you agree to change it in a separate PR? as it is something that was added with the initial release of the integration.

This can be another PR (though I would rather it not be since we are changing all the lines that need to be fixed in this change anyway, and it is a UI design bug); there is no need to have an active link to the page you are on and the text should tell the user where they are without having to read additional text. This is something that we discussed internally a few months back (2024-10-30). example comparison.

elasticmachine · 2025-04-16T18:14:58Z

💚 Build Succeeded

Buildkite Build
Commit: 32baa31

History

💚 Build #24599 succeeded fccdd38
💔 Build #24585 failed fccdd38
💔 Build #24583 failed 10a4cb8

cc @chemamartinez

elastic-sonarqube · 2025-04-16T18:15:07Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

efd6

Approving, but please reconsider the UI fix.

elastic-vault-github-plugin-prod · 2025-04-16T22:29:33Z

Package abnormal_security - 1.6.0 containing this change is available at https://epr.elastic.co/package/abnormal_security/1.6.0/

chemamartinez · 2025-04-17T12:53:36Z

Approving, but please reconsider the UI fix.

Auto-merge was enabled but I will open a new PR with your suggestion.

chemamartinez added enhancement New feature or request dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Integration:abnormal_security Abnormal AI labels Apr 9, 2025

chemamartinez self-assigned this Apr 9, 2025

chemamartinez marked this pull request as ready for review April 9, 2025 07:38

chemamartinez requested a review from a team as a code owner April 9, 2025 07:38

chemamartinez added 6 commits April 9, 2025 09:54

New data stream ai_security_mailbox_not_analyzed

e1db2f6

Small adjustments to all data streams

5a83996

Bump version

10f2baa

Add AI Mailbox Not Analyzed dashboard

833a644

Fix typo in docs

b537a6a

Update PR link

fccdd38

chemamartinez force-pushed the 13000-abnormal_security-not_analyzed branch from 10a4cb8 to fccdd38 Compare April 9, 2025 07:54

chemamartinez enabled auto-merge (squash) April 9, 2025 13:10

efd6 reviewed Apr 10, 2025

View reviewed changes

Add Dan's requested changes

32baa31

chemamartinez requested a review from efd6 April 16, 2025 17:49

efd6 approved these changes Apr 16, 2025

View reviewed changes

chemamartinez merged commit 3e63dee into elastic:main Apr 16, 2025
7 checks passed

chemamartinez deleted the 13000-abnormal_security-not_analyzed branch April 22, 2025 09:34

chemamartinez mentioned this pull request Apr 22, 2025

abnormal_security: fix navigation between dashboards in tables of content #13643

Merged

5 tasks

	bytes(resp.Body).decode_json().as(body, has(body.results) && size(body.results) > 0 ?
	resp.Body.decode_json().as(body, has(body.results) && size(body.results) > 0 ?

	"message": "GET:"+(
	"message": "GET "+state.url.trim_right("/") + "/v1/abuse_mailbox/not_analyzed:"+(

		"start": [string(state.start_time)],
		"end": [string(state.end_time)],

abnormal_security: add support for Not Analyzed Messages data stream #13483

abnormal_security: add support for Not Analyzed Messages data stream #13483

Uh oh!

Conversation

chemamartinez commented Apr 9, 2025

Proposed commit message

Checklist

Related issues

Screenshots

Uh oh!

elasticmachine commented Apr 9, 2025

Uh oh!

chemamartinez commented Apr 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Apr 16, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Apr 16, 2025

Quality Gate passed

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Apr 16, 2025

Uh oh!

chemamartinez commented Apr 17, 2025

Uh oh!

Uh oh!

efd6 Apr 16, 2025 •

edited

Loading