Traefik 2.x access-log support #9131

ltflb-bgdi · 2024-02-12T20:34:59Z

Proposed commit message

Traefik-access-log integration was done with traefik v1.6 which went EOL 2018. Since traefik 2.x the (json) log format changed.
Thus using the existing traefik integration is not working properly. As well, only few fields are parsed and some important ones are missing.

For this reason I have done the following improvements:

Refactored field structure according to traefik 2.x format (json)
Parsed more fields
Improved ecs compatibility

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Json log tests are failing due to http headers. These headers are mapped as objects as already done in apm integration. Expected logs correct but elastic-package test fails with missing header field mapping comment. This seems to me like a limitation/bug of the elastic-package tool and should be fixed separately.

How to test this PR locally

elastic-package stack up
cd /packages/traefik
elastic-package test

Related issues

Relates [traefik] Traefik v2 Access Logs format #9117, [Traefik 1.11.0] Errors in Ingest Pipeline logs-traefik.access-1.11.0 #8886

Screenshots

LaZyDK · 2024-02-22T17:12:28Z

I'm looking forward to this one.

jamiehynds · 2024-02-28T12:14:00Z

@SubhrataK can someone from your team please review?

muthu-mps · 2024-03-05T11:12:31Z

@SubhrataK can someone from your team please review?

Thanks @jamiehynds - We will look into it.

milan-elastic · 2024-03-07T05:57:16Z

packages/traefik/data_stream/access/_dev/test/pipeline/test-format-common.log-expected.json

@@ -11,7 +11,7 @@
                ],
                "created": "2020-04-28T11:07:58.223Z",
                "duration": 2000000,
-                "ingested": "2023-10-15T20:29:57.434126924Z",
+                "ingested": "2024-02-14T16:03:17.172004634Z",
                "kind": "event",
                "original": "192.168.33.1 - - [02/Oct/2017:20:22:07 +0000] \"GET /ui/favicons/favicon-16x16.png HTTP/1.1\" 304 0 \"http://example.com/login\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36\" 262 \"Host-host-1\" \"http://172.19.0.3:5601\" 2ms",


Please mask the sensitive information such like IPs, domain name etc. These sensitive information shouldn't be shipped in package. please check the same for other places as well.

milan-elastic · 2024-03-07T06:03:28Z

packages/traefik/changelog.yml

@@ -1,3 +1,8 @@
+- version: "2.0.0"
+  changes:
+    - description: Support traefik v2.x access-logs


Minor nit: Please end the description with "." (fullstop)

Done for all descriptions

milan-elastic · 2024-03-07T06:11:12Z

packages/traefik/_dev/deploy/variants.yml

@@ -3,4 +3,6 @@ variants:
    SERVICE_VERSION: 1.6
  "v1.7":


Suggestion: If we are keeping 1.7 version in variants then can we also update the Compatibility section by saying that "The Traefik datasets were tested with Traefik 1.6, 1.7 and 2.9" after testing?

milan-elastic · 2024-03-07T06:31:21Z

packages/traefik/data_stream/access/_dev/test/pipeline/test-format-common.log-expected.json

+                    },
+                    "service": {
+                        "url": {
+                            "original": "http://172.19.0.3:5601"


Based on the description of the field and logs pattern it seems it should be mapped with ECS field called service.address.

milan-elastic · 2024-03-07T06:47:32Z

packages/traefik/data_stream/access/elasticsearch/ingest_pipeline/format-json.yml

+  - set:
+      field: observer.type
+      value: proxy
+


Can you add the null remover script? As null value shouldn't be ingested. For reference : https://github.com/elastic/integrations/blob/main/packages/apache/data_stream/access/elasticsearch/ingest_pipeline/default.yml#L171-L194

milan-elastic · 2024-03-07T06:54:39Z

packages/traefik/data_stream/access/_dev/test/pipeline/test-format-json.log-expected.json

@@ -1,7 +1,7 @@
 {
    "expected": [
        {
-            "@timestamp": "2021-03-16T18:56:54Z",
+            "@timestamp": "2021-03-16T18:56:54.735539596Z",
            "ecs": {


We have updated the ECS version from 8.5.1 to 8.11.0 in dependancy hence please update this to 8.11.0 as well by setting up the processor in pipeline. For reference: https://github.com/elastic/integrations/blob/main/packages/apache/data_stream/access/elasticsearch/ingest_pipeline/default.yml#L10C2-L12C21

muthu-mps · 2024-03-13T05:11:57Z

/test

muthu-mps · 2024-03-13T05:27:21Z

@ltflb-bgdi,

The docs/README.md file should be generated. Perform the changes to README file in the _dev/build/docs folder and run the elastic-package build command to update the docs/README.md file.
Build fails for this reason
Error: checking package failed: checking readme files are up-to-date failed: files do not match

ltflb-bgdi · 2024-03-13T07:49:28Z

@ltflb-bgdi,

The docs/README.md file should be generated. Perform the changes to README file in the _dev/build/docs folder and run the elastic-package build command to update the docs/README.md file.

Build fails for this reason
Error: checking package failed: checking readme files are up-to-date failed: files do not match

Done

muthu-mps · 2024-03-13T07:51:30Z

/test

muthu-mps · 2024-05-07T04:44:21Z

We need to look into the changes required in the metrics for the new version. Primarily we need to look at how to collect the metrics with the newer version with multiple options provided. Let me comeback to you on this.

As a passive observer on this that is interested in using the logging data stream, perhaps there is an option to merge the logging change separately from health/metric changes.

@andrewkroh - My only concern is what happens if the log is pulled from the newer version and the metrics rely on the older version. Let is evaluate this scenario if nothing breaks then we should be good to merge this PR. Please add your thoughts around this.

@milan-elastic - Can we validate this PR changes by updating the integrations from the Traefik 1.x version to the 2.x version.

andrewkroh · 2024-05-07T11:16:55Z

My only concern is what happens if the log is pulled from the newer version and the metrics rely on the older version

That's true. The /health data stream won't work. I imaging there isn't anyone using Traefik 1.x anymore. It entered end-of-support¹ close to 2.5 years ago (with no security updates) and now 3.x is out. I would take the stance that this integration should no longer try to support 1.x and it drops support for it by removing the heath data stream and incrementing the packages major version.

Traefik has several ways of getting rich metrics from it like prometheus, otel, and statd. (Personally I pull from the prometheus endpoint). You could plan a new metrics data stream that pulls from prometheus/statds or perhaps could document that metrics can be sent to the APM server's OTLP listener².

muthu-mps · 2024-05-07T13:01:42Z

My only concern is what happens if the log is pulled from the newer version and the metrics rely on the older version

That's true. The /health data stream won't work. I imaging there isn't anyone using Traefik 1.x anymore. It entered end-of-support1 close to 2.5 years ago (with no security updates) and now 3.x is out. I would take the stance that this integration should no longer try to support 1.x and it drops support for it by removing the heath data stream and incrementing the packages major version.

Traefik has several ways of getting rich metrics from it like prometheus, otel, and statd. (Personally I pull from the prometheus endpoint). You could plan a new metrics data stream that pulls from prometheus/statds or perhaps could document that metrics can be sent to the APM server's OTLP listener2.

Footnotes

https://doc.traefik.io/traefik/deprecation/releases/ ↩

https://www.elastic.co/guide/en/observability/current/apm-open-telemetry.html ↩

Thanks Andrew! Valid point. Since the native support for processing metrics is already unavailable we can deprecate the current data stream and add a new data stream for processing the metrics. Let me create an issue.

milan-elastic · 2024-05-08T13:37:41Z

@milan-elastic - Can we validate this PR changes by updating the integrations from the Traefik 1.x version to the 2.x version.

Findings

As per @muthu-mps comment I have tested out Traefik upgrade scenario from 1.x to 2.x. Here are the scenarios tested

Verified that the Traefix 1.x format logs are still parsed by the pipeline in the Traefik integration version 2.0.0
Verified that the Traefix 2.x format logs are still parsed by the pipeline in the Traefik integration version 2.0.0
- I did find some fields that are no longer supported in Traefik integration version 2.0.0 like traefik.access.backend_url and traefik.access.frontend_name . These fields are now renamed to traefik.access.service.address and traefik.access.router.name respectively.
Verified that there are no errors or conflicts while the integration is upgraded to 2.0.0 version
Verified by loading the dashboards and visualizations after upgrading the integration to the 2.x major release and checked that there are no errors in it.

Major issue

I understand that we are saying the user to upgrade to the newer versions of Traefik but this would lead to breaking the health data stream as the /health endpoint is no longer accessible just as Andrew said. Tried accessing the endpoint on Traefik 1.7 initially and there it was accessible and we are able to fetch the metrics but in Traefik 2.5 this is not working. You can find the setup links here for both the versions

Minor nits

Need to update the sample_event.json as I can see some of the empty fields in it. We have already included the null script, so we should once generate the sample_event to check if it is working

muthu-mps · 2024-05-14T04:23:59Z

As we plan to release the log stream in this iteration and the metrics stream in the next iteration. We need to update the integration with the below mentioned changes.

Steps

Remove the metrics data stream from the policy template. Ensures that we are not adding unsupported data stream.
Update the documentation to have a mention that the integrations supports log collection with the Traefik supported versions and the support for metrics collection will be added.
Documentation should remove the metric data stream references.
The current dashboard supports only the logs. Verify the data is loading properly for logs after making the above changes.

@ltflb-bgdi - Please let us know, If you need any help in making the above changes. If you are okay, we can perform the changes to the PR.

ltflb-bgdi · 2024-05-14T14:39:49Z

Update the documentation to have a mention that the integrations supports log collection with the Traefik supported versions and the support for metrics collection will be added.

Documentation should remove the metric data stream references.

@muthu-mps Feel free to implement your proposed changes. If you want me to do it, let me know.

muthu-mps · 2024-05-15T05:17:43Z

Update the documentation to have a mention that the integrations supports log collection with the Traefik supported versions and the support for metrics collection will be added.

Documentation should remove the metric data stream references.

@muthu-mps Feel free to implement your proposed changes. If you want me to do it, let me know.

Thanks @ltflb-bgdi!

@milan-elastic - Please make the above suggested changes to this PR. Let me know if you have any questions.

milan-elastic · 2024-05-15T14:36:06Z

@ltflb-bgdi can you please provide me an access to your project so I can write into your branch?
cc: @muthu-mps

ltflb-bgdi · 2024-05-17T09:25:29Z

@ltflb-bgdi can you please provide me an access to your project so I can write into your branch? cc: @muthu-mps

Done

muthu-mps · 2024-05-20T03:39:49Z

/test

muthu-mps · 2024-05-20T07:19:58Z

/test

muthu-mps · 2024-05-20T08:07:04Z

/test

elasticmachine · 2024-05-20T08:23:42Z

💚 Build Succeeded

Buildkite Build
Commit: 189c547

History

💔 Build #11679 failed a10f9a0
💔 Build #11673 failed 48f6ab4
💚 Build #10464 succeeded 32f44d8
💔 Build #10280 failed 350af9d
💔 Build #10177 failed db48c00

elastic-sonarqube · 2024-05-20T08:23:43Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
91.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

milan-elastic

LGTM!

muthu-mps · 2024-05-20T10:27:44Z

@andrewkroh - We have updated the policy template to drop support for metrics input and the support for metrics will get revamped and added in the next iterations. The documentation is also updated.
Do you have any final thoughts before merging the PR?

andrewkroh

🚢

ltflb-bgdi · 2024-06-10T11:26:12Z

Please let me know by when we can expect the last approval and merge. Thanks.

muthu-mps

LGTM!

elasticmachine · 2024-06-11T11:06:03Z

Package traefik - 2.0.0 containing this change is available at https://epr.elastic.co/search?package=traefik

ltflb-bgdi requested a review from a team as a code owner February 12, 2024 20:35

ltflb-bgdi marked this pull request as draft February 12, 2024 20:35

ltflb-bgdi force-pushed the feat-BGDIINF_SB-3114-traefik branch 9 times, most recently from 7d4fd2f to 9da289e Compare February 15, 2024 10:19

ltflb-bgdi marked this pull request as ready for review February 16, 2024 16:30

jamiehynds added Integration:traefik Traefik Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Feb 28, 2024

lalit-satapathy requested review from milan-elastic and muthu-mps March 5, 2024 10:54

milan-elastic reviewed Mar 7, 2024

View reviewed changes

ltflb-bgdi requested a review from milan-elastic March 11, 2024 17:28

muthu-mps mentioned this pull request May 9, 2024

Traefik Integration Migrations #9821

Open

muthu-mps added 3 commits May 20, 2024 08:01

Update Documentation

7ba4cbf

Update Documentation

76932c8

Remove metrics input

48f6ab4

muthu-mps and others added 3 commits May 20, 2024 10:10

Update README.md

9e9852f

Address Lint error

5a1fb1b

remove system tests

a10f9a0

reverting the system test changes

189c547

milan-elastic approved these changes May 20, 2024

View reviewed changes

andrewkroh approved these changes May 20, 2024

View reviewed changes

muthu-mps approved these changes Jun 11, 2024

View reviewed changes

muthu-mps merged commit 7a68798 into elastic:main Jun 11, 2024

muthu-mps deleted the feat-BGDIINF_SB-3114-traefik branch June 11, 2024 10:51

Traefik 2.x access-log support #9131

Traefik 2.x access-log support #9131

Uh oh!

Conversation

ltflb-bgdi commented Feb 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

LaZyDK commented Feb 22, 2024

Uh oh!

jamiehynds commented Feb 28, 2024

Uh oh!

muthu-mps commented Mar 5, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

muthu-mps commented Mar 13, 2024

Uh oh!

muthu-mps commented Mar 13, 2024

Uh oh!

ltflb-bgdi commented Mar 13, 2024

Uh oh!

muthu-mps commented Mar 13, 2024

Uh oh!

muthu-mps commented May 7, 2024

Uh oh!

andrewkroh commented May 7, 2024

Footnotes

Uh oh!

muthu-mps commented May 7, 2024

Footnotes

Uh oh!

milan-elastic commented May 8, 2024

Findings

Major issue

Minor nits

Uh oh!

muthu-mps commented May 14, 2024

Steps

Uh oh!

ltflb-bgdi commented May 14, 2024

Uh oh!

muthu-mps commented May 15, 2024

Uh oh!

milan-elastic commented May 15, 2024

Uh oh!

ltflb-bgdi commented May 17, 2024

Uh oh!

muthu-mps commented May 20, 2024

Uh oh!

muthu-mps commented May 20, 2024

Uh oh!

muthu-mps commented May 20, 2024

Uh oh!

elasticmachine commented May 20, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented May 20, 2024

Quality Gate passed

Uh oh!

ltflb-bgdi commented Feb 12, 2024 •

edited

Loading