Skip to content

[apache_tomcat] Improve access pipeline performance by split grok and switch to dissect #8723

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Dec 19, 2023
Merged

[apache_tomcat] Improve access pipeline performance by split grok and switch to dissect #8723

merged 8 commits into from
Dec 19, 2023

Conversation

SimonKoetting
Copy link
Contributor

@SimonKoetting SimonKoetting commented Dec 13, 2023

Restructuring apache_tomcat.access ingest pipeline to use dissect as first step and split grok into multiple processors to improve perfomance.
On test-env the ingest performance was more den doubled, from ~0.899ms per Document (average) to ~0.253ms per Document (average)

image

@SimonKoetting SimonKoetting requested a review from a team as a code owner December 13, 2023 12:37
@SimonKoetting SimonKoetting changed the title Improve access pipeline performance by split grok and switch to dissect [apache_tomcat] Improve access pipeline performance by split grok and switch to dissect Dec 13, 2023
@elasticmachine
Copy link

elasticmachine commented Dec 13, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-12-14T11:41:29.938+0000

  • Duration: 24 min 29 sec

Test stats 🧪

Test Results
Failed 0
Passed 54
Skipped 0
Total 54

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Dec 13, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (9/9) 💚
Files 100.0% (9/9) 💚 3.888
Classes 100.0% (9/9) 💚 3.888
Methods 97.561% (80/82) 👍 5.255
Lines 82.776% (990/1196) 👎 -5.968
Conditionals 100.0% (0/0) 💚

@elasticmachine
Copy link

elasticmachine commented Dec 13, 2023

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@harnish-crest-data
Copy link
Contributor

@SimonAtElastic, Can you please confirm as per your latest commit, the ingest performance is still ~304ms (average)?

@SimonKoetting
Copy link
Contributor Author

SimonKoetting commented Dec 15, 2023

@SimonAtElastic, Can you please confirm as per your latest commit, the ingest performance is still ~304ms (average)?

Not exactly, i now average ~0.25ms, so even improved :) I did the seperation of the grok before the dissect and then i got a slight improvement but after the dissect its performing better when combined.

Copy link
Contributor

@harnish-crest-data harnish-crest-data left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 🚀

Copy link
Member

@ishleenk17 ishleenk17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@ishleenk17 ishleenk17 merged commit e1354c1 into elastic:main Dec 19, 2023
@elasticmachine
Copy link

Package apache_tomcat - 1.2.0 containing this change is available at https://epr.elastic.co/search?package=apache_tomcat

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants