[O11y][Apache] Update grok pattern for access and error log data streams #10228

harnish-crest-data · 2024-06-24T13:11:27Z

Enhancement

Proposed commit message

Update grok pattern to support combined log format for access data stream.
- %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
- %A:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
- %h:%p %l %u %t \"%{req}i %U %H\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
Update grok pattern to support new logs for error data stream. Refer

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Closes Apache integration Provided Grok expressions do not match field value #8512

elasticmachine · 2024-06-24T13:26:33Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

…pache-access-logs Conflicts: packages/apache/_dev/build/docs/README.md packages/apache/changelog.yml packages/apache/data_stream/access/_dev/test/pipeline/test-access-basic.log-expected.json packages/apache/data_stream/access/_dev/test/pipeline/test-access-darwin.log-expected.json packages/apache/data_stream/access/_dev/test/pipeline/test-access-ssl-request.log-expected.json packages/apache/data_stream/access/_dev/test/pipeline/test-access-ubuntu.log-expected.json packages/apache/data_stream/access/_dev/test/pipeline/test-access-vhost.log-expected.json packages/apache/data_stream/access/fields/ecs.yml packages/apache/data_stream/error/_dev/test/pipeline/test-error-basic.log-expected.json packages/apache/docs/README.md

kush-elastic

LGTM!

muthu-mps · 2024-07-17T06:05:19Z

packages/apache/data_stream/access/elasticsearch/ingest_pipeline/default.yml

+      ignore_missing: true
+      on_failure:
+          - set:
+              field: tmp_host


Are we removing the tmp_host field after processing is done?

No, it will be automatically removed by null processor as we are setting it's value to empty string. The reason behind this thing is, we need to do swapping up the source.address value!

muthu-mps · 2024-07-17T06:08:21Z

packages/apache/data_stream/error/_dev/test/pipeline/test-error-basic.log

@@ -2,3 +2,4 @@
 [Mon Dec 26 16:15:55.103786 2016] [core:notice] [pid 11379] AH00094: Command line: '/usr/local/Cellar/httpd24/2.4.23_2/bin/httpd'
 [Fri Sep 09 10:42:29.902022 2011] [core:error] [pid 35708:tid 4328636416] [client 89.160.20.156] File does not exist: /usr/local/apache2/htdocs/favicon.ico
 [Thu Jun 27 06:58:09.169510 2019] [include:warn] [pid 15934] [client 89.160.20.156:12345] AH01374: mod_include: Options +Includes (or IncludesNoExec) wasn't set, INCLUDES filter removed: /test.html
+AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message


Is this a valid error log message? Does this entry doesn't log the timestamp?

Please checkout this quote that I have already attached in the description. This is warning log that is coming in the error log file and this log is the main reason of failure in the grok pattern!

muthu-mps · 2024-07-17T06:12:51Z

packages/apache/data_stream/error/_dev/test/pipeline/test-error-basic.log-expected.json

@@ -184,6 +184,24 @@
            "tags": [
                "preserve_original_event"
            ]
+        },
+        {


Did we try loading this document without @timestamp field in the dashboard? What happens if the user is trying to filter data for a specific time range in discover or in the dashboard?

Yes we have tried this and if there are no timestamp present, it will take the doc ingestion time.

muthu-mps · 2024-07-17T06:14:56Z

Are we supporting this log format as well?

harnish-crest-data · 2024-07-17T07:07:13Z

Are we supporting this log format as well?

Yes we are supporting this log format. Let me update the same in pipeline tests!

muthu-mps · 2024-07-17T09:25:18Z

Are we supporting this log format as well?

Yes we are supporting this log format. Let me update the same in pipeline tests!

Can you update the sample log in the pipeline test?

harnish-crest-data · 2024-07-17T09:29:44Z

Are we supporting this log format as well?

Yes we are supporting this log format. Let me update the same in pipeline tests!

Can you update the sample log in the pipeline test?

Updated, thanks!

muthu-mps · 2024-07-17T10:58:20Z

packages/apache/_dev/build/docs/README.md

@@ -23,30 +25,49 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
 Supported format for the access logs are:

 - [Common Log Format](https://en.wikipedia.org/wiki/Common_Log_Format)


Please change the reference link to point to the official Apache documentation.

Verify the link below, if its not specific to a release version we can include the below link for reference.
common log format

Updated, thanks!

packages/apache/_dev/build/docs/README.md

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

muthu-mps

LGTM!

…pache-access-logs Conflicts: packages/apache/changelog.yml

elasticmachine · 2024-07-22T06:35:41Z

💚 Build Succeeded

Buildkite Build
Commit: b9552e8

History

💚 Build #13739 succeeded ca9df19
💚 Build #13727 succeeded c737f86
💚 Build #13419 succeeded b78f947
💔 Build #13418 failed 6abd9b4
💚 Build #13359 succeeded d26dea7
💚 Build #12780 succeeded 457abe9

cc @harnish-elastic

elastic-sonarqube · 2024-07-22T06:35:43Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
94.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-07-22T07:48:02Z

Package apache - 1.23.0 containing this change is available at https://epr.elastic.co/search?package=apache

Update grok pattern to support combined log format

457abe9

harnish-crest-data self-assigned this Jun 24, 2024

harnish-crest-data mentioned this pull request Jun 24, 2024

Apache integration Provided Grok expressions do not match field value #8512

Closed

update grok pattern for error logs

d26dea7

harnish-crest-data marked this pull request as ready for review July 8, 2024 10:42

harnish-crest-data requested a review from a team as a code owner July 8, 2024 10:42

harnish-crest-data changed the title ~~[O11y][Apache] Update grok pattern to support combined log format~~ [O11y][Apache] Update grok pattern for access and error log data streams Jul 9, 2024

harnish-crest-data added 2 commits July 9, 2024 17:57

pipeline tests generate

b78f947

harnish-crest-data requested a review from kush-elastic July 15, 2024 07:02

kush-elastic approved these changes Jul 15, 2024

View reviewed changes

harnish-crest-data requested a review from muthu-mps July 16, 2024 06:10

muthu-mps reviewed Jul 17, 2024

View reviewed changes

harnish-crest-data requested a review from muthu-mps July 17, 2024 07:15

update pipeline tests

c737f86

muthu-mps reviewed Jul 17, 2024

View reviewed changes

harnish-crest-data and others added 4 commits July 17, 2024 18:13

Update packages/apache/_dev/build/docs/README.md

0087e75

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

Update packages/apache/_dev/build/docs/README.md

4444eb1

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

Update packages/apache/_dev/build/docs/README.md

06993ab

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

address review comments

ca9df19

harnish-crest-data requested a review from muthu-mps July 17, 2024 12:50

muthu-mps approved these changes Jul 17, 2024

View reviewed changes

andrewkroh added Integration:apache Apache HTTP Server Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Jul 19, 2024

Merge branch 'main' of github.com:harnish-elastic/integrations into a…

b9552e8

…pache-access-logs Conflicts: packages/apache/changelog.yml

harnish-crest-data merged commit 35cd691 into elastic:main Jul 22, 2024
5 checks passed

Alphayeeeet mentioned this pull request Jul 22, 2024

[Apache.access] Changed source and destination fields in a backward-compatible manner (ECS) #10416

Closed

4 tasks

		@@ -23,30 +25,49 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
		Supported format for the access logs are:

		- [Common Log Format](https://en.wikipedia.org/wiki/Common_Log_Format)

[O11y][Apache] Update grok pattern for access and error log data streams #10228

[O11y][Apache] Update grok pattern for access and error log data streams #10228

Uh oh!

Conversation

harnish-crest-data commented Jun 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Related issues

Uh oh!

elasticmachine commented Jun 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

kush-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

muthu-mps commented Jul 17, 2024

Uh oh!

harnish-crest-data commented Jul 17, 2024

Uh oh!

muthu-mps commented Jul 17, 2024

Uh oh!

harnish-crest-data commented Jul 17, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

muthu-mps left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 22, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Jul 22, 2024

Quality Gate passed

Uh oh!

Uh oh!

elasticmachine commented Jul 22, 2024

Uh oh!

Uh oh!

harnish-crest-data commented Jun 24, 2024 •

edited

Loading

elasticmachine commented Jun 24, 2024 •

edited

Loading