Integration of Microsoft SQL Server - Audit Events #2009

r00tu53r · 2021-10-25T05:43:57Z

What does this PR do?

Build a new integration package for Microsoft SQL Server Audit Events. (#1635).

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

Author's Checklist

Still working on adding visualizations and saved searches etc.

How to test this PR locally

elastic-package test pipeline

Related issues

Closes MSSQL Audit Integration #1635

Screenshots

Define split ingest processor to transform raw data into new line separated entries

Fix event_time from date to text due to agent side parsing errors

- Add all fields - Add conversions where necessasry

elasticmachine · 2021-10-25T05:52:35Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-12-08T02:33:16.161+0000
Duration: 12 min 23 sec
Commit: ce2bc9a

Test stats 🧪

Test	Results
Failed	0
Passed	4
Skipped	0
Total	4

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

P1llus · 2021-10-25T11:37:10Z

There was an initial discussion on this on slack, awaiting further changes to be pushed before doing another review.

- Change field types to numbers for those which can be queried by range - Set kv target directly to sqlserver.audit and remove _temp - Update README

- Add event_id as configurable input - Handle tags and preserve original event - Add processors

elasticmachine · 2021-10-26T06:01:38Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

packages/microsoft_sqlserver/_dev/build/docs/README.md

...ages/microsoft_sqlserver/data_stream/audit/_dev/test/pipeline/test-events.json-expected.json

packages/microsoft_sqlserver/data_stream/audit/fields/fields.yml

packages/microsoft_sqlserver/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

andrewkroh · 2021-12-06T23:09:11Z

packages/microsoft_sqlserver/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+    target_field: process
+    ignore_missing: true
+- rename:
+    field: winlog.user


It looks like this might be the process that wrote the event log rather than the SQL user that performed the action. So I would leave it unmodified.

I would expect user.* to be based on the user that's logged into the DB. I think that means server_principal_* is used to populate user.* and target_server_principal_* gets copied into user.target.*.

I have addressed these too. From what I understand the server_principal_sid for a domain account is stored in the encoded binary form in SQL Server so it probably isn't sending target_server_principal_sid (I haven't seen it in my tests) as they are the same. I've set it anyway in case we receive it, we'll set it.

So these are not the standard Windows SID strings (S-R-I-S-S format), but are a SQS server SID in varbinary format? Perhaps it we should retain the original "principal" keys to have some traceability on where the values of user.* came from.

If we never retain the original principal fields then they should be removed from the mappings.

The value in these depend on the type of login / account. SQL Server can be configured to allow logins in Mixed Mode or Windows Authentication Mode. Microsoft recommends to use Windows authentication as it is more secure. In mixed mode the user could be a SQL server user in which case the SID is a 16 byte varbinary specific to SQL Server version. But if the login is tied to Windows authentication then SQL server hands off the authentication to Windows domain controller. For this case the SID is windows SID but converted into binary format. There seems to be functions available to convert them back to string representation (S-1-x-x-x) in SQL server.

From what I understand, the value could be the original principal SID but in a different format. It could be converted between varbinary and text formats in SQL Server.

packages/microsoft_sqlserver/manifest.yml

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

…ngest_pipeline/default.yml Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

* Update fields from text to keyword where applicable * Change host.mac format to ECS recommendation

…leet integrations

P1llus

LGTM! Great work on the package, sorry it tok so long. Did a lot of the reviewing outside of Github as well

* Set SPN and Target SPN to user.* and user.target.* ECS fields * Cleanup

r00tu53r · 2021-12-08T02:36:19Z

@andrewkroh Thank you for your comments. I've addressed them all. I'll wait for you to take another look.

r00tu53r · 2021-12-08T02:37:07Z

LGTM! Great work on the package, sorry it tok so long. Did a lot of the reviewing outside of Github as well

Thank you @P1llus for your inputs and help with this integration! No worries about the timelines.

andrewkroh

I think the organization of the pipeline needs changed if events other than 33205 are expected to be processed. Everything else LGTM.

r00tu53r · 2021-12-14T01:46:16Z

I think the organization of the pipeline needs changed if events other than 33205 are expected to be processed. Everything else LGTM.

Once the SQL Server has been configured to log audit events to event log and has no issues w.r.t permissions etc all events are logged with id 33205 starting from MS SQL Server 2008 as stated here.

There are some audit related events reaching other IDs like 33203, 33204, 33210 etc (stated here). These are for cases where audit could not be performed due to permission issues etc.

r00tu53r added 11 commits October 21, 2021 14:20

initial commit - working sending raw message

537646f

Add field definitions for ecs and winlog

aa1a2a0

Define fields for processing raw data

8908773

Define split ingest processor to transform raw data into new line separated entries

Add README with field template

e79abd8

Add sqlserver.audit fields and documentation

20a5371

Add kv to split param1 to key-values

004181e

Fix event_time from date to text due to agent side parsing errors

Pass pipeline tests

0555660

- Add all fields - Add conversions where necessasry

Change type for those that are integers

956b221

Set timestamp based on audit event_time

bdb18ce

Remove non-existent fields for SQL server Audit event

38552a5

Update README - for field changes

dceeb3b

r00tu53r added enhancement New feature or request draft Draft labels Oct 25, 2021

r00tu53r requested a review from P1llus October 25, 2021 05:43

r00tu53r self-assigned this Oct 25, 2021

Update changelog

3ac3c6e

Format JSON

136b479

r00tu53r added 7 commits October 26, 2021 11:32

Update README with information on enabling auditing for SQL server

0234c40

Change field types and pipeline

4e83907

- Change field types to numbers for those which can be queried by range - Set kv target directly to sqlserver.audit and remove _temp - Update README

Handle preserve original event tag

3a75fc4

Change audit manifest

aed8c74

- Add event_id as configurable input - Handle tags and preserve original event - Add processors

Update pipeline and tests

bf1d0b2

set event.category to database

92c631f

fix formatting

584d7ed

P1llus added the Team:Security-External Integrations label Oct 26, 2021

P1llus added the 7.16 candidate label Oct 26, 2021

Fix test - add event.category

f06047f

r00tu53r added 2 commits November 29, 2021 20:48

Fix formatting

c7afa3e

Set kibana version

34a678c

andrewkroh requested a review from a team November 30, 2021 16:11

andrewkroh reviewed Dec 6, 2021

View reviewed changes

r00tu53r and others added 14 commits December 6, 2021 16:20

Update packages/microsoft_sqlserver/_dev/build/docs/README.md

a5670ae

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update packages/microsoft_sqlserver/manifest.yml

1dcc76a

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update packages/microsoft_sqlserver/manifest.yml

7cdc500

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update packages/microsoft_sqlserver/data_stream/audit/elasticsearch/i…

945338a

…ngest_pipeline/default.yml Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Fix field types and add host.mac processor

4fb4f06

* Update fields from text to keyword where applicable * Change host.mac format to ECS recommendation

Change client_ip from text to keyword

8bfbf93

Consolidate remove fields under one processor

a9550db

Use dissect instead of grok to extract statement

0808252

Remove event.ingested as it is added through final_pipeline for all f…

64e12e8

…leet integrations

Remove setting event.code as its set by winlog input

492ac25

Remove unneccessary type creation

2c4e17b

Refactor painless script

96e5218

Convert duration_milliseconds to ns and set to event.duration

9688d3a

Update test expected output

4b6ccae

P1llus approved these changes Dec 7, 2021

View reviewed changes

Set user fields

ce2bc9a

* Set SPN and Target SPN to user.* and user.target.* ECS fields * Cleanup

epixa added 8.1 candidate and removed 7.16 candidate labels Dec 8, 2021

jamiehynds mentioned this pull request Dec 9, 2021

New Module Request : MSSQL Audit Module elastic/beats#20001

Closed

9 tasks

andrewkroh reviewed Dec 13, 2021

View reviewed changes

r00tu53r merged commit f1edb32 into elastic:master Dec 14, 2021

andrewkroh added Integration:microsoft_sqlserver Microsoft SQL Server New Integration Issue or pull request for creating a new integration package. labels Aug 13, 2024

Integration of Microsoft SQL Server - Audit Events #2009

Integration of Microsoft SQL Server - Audit Events #2009

Uh oh!

Conversation

r00tu53r commented Oct 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Oct 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

P1llus commented Oct 25, 2021

Uh oh!

elasticmachine commented Oct 26, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewkroh Dec 6, 2021

Choose a reason for hiding this comment

Uh oh!

r00tu53r Dec 8, 2021

Choose a reason for hiding this comment

Uh oh!

andrewkroh Dec 13, 2021

Choose a reason for hiding this comment

Uh oh!

r00tu53r Dec 14, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

P1llus left a comment

Choose a reason for hiding this comment

Uh oh!

r00tu53r commented Dec 8, 2021

Uh oh!

r00tu53r commented Dec 8, 2021

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

r00tu53r commented Dec 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

r00tu53r commented Oct 25, 2021 •

edited

Loading

elasticmachine commented Oct 25, 2021 •

edited

Loading

r00tu53r commented Dec 14, 2021 •

edited

Loading