-
Notifications
You must be signed in to change notification settings - Fork 535
Clarify parsing multiline messages (#10212) #1927
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
@@ -19,13 +19,13 @@ The Multiline parser engine exposes two ways to configure and use the feature: | |||||||||||||
Fluent Bit exposes certain pre-configured parsers (built-in) to solve specific | ||||||||||||||
multiline parser cases. For example: | ||||||||||||||
|
||||||||||||||
| Parser | Description | | ||||||||||||||
| ------ | ----------- | | ||||||||||||||
| `docker` | Process a log entry generated by a Docker container engine. This parser supports the concatenation of log entries split by Docker. | | ||||||||||||||
| `cri` | Process a log entry generated by CRI-O container engine. Like the `docker` parser, it supports concatenation of log entries | | ||||||||||||||
| `go` | Process log entries generated by a Go based language application and perform concatenation if multiline messages are detected. | | ||||||||||||||
| `python` | Process log entries generated by a Python based language application and perform concatenation if multiline messages are detected. | | ||||||||||||||
| `java` | Process log entries generated by a Google Cloud Java language application and perform concatenation if multiline messages are detected. | | ||||||||||||||
| Parser | Description | | ||||||||||||||
|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||||||||||||||
| `docker` | Process a log entry generated by a Docker container engine. This This parser supports the concatenation of large log entries split by Docker. If you use this parser, and you also want to concatenate loglines like stacktraces, you can add the [multiline filter](../../pipeline/filters/multiline-stacktrace.md) to specify additional parsers | | ||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Think there's a typo here with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||
| `cri` | Process a log entry generated by CRI-O container engine. Like the `docker` parser, it supports concatenation of log entries | | ||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||
| `go` | Process log entries generated by a Go based language application and perform concatenation if multiline messages are detected. | | ||||||||||||||
| `python` | Process log entries generated by a Python based language application and perform concatenation if multiline messages are detected. | | ||||||||||||||
| `java` | Process log entries generated by a Google Cloud Java language application and perform concatenation if multiline messages are detected. | | ||||||||||||||
Comment on lines
+26
to
+28
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||
|
||||||||||||||
### Configurable multiline parsers | ||||||||||||||
|
||||||||||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -17,7 +17,7 @@ Along with multiline filters, you can enable one of the following built-in Fluen | |
When using this filter: | ||
|
||
- The usage of this filter depends on a previous configuration of a [multiline parser](../../administration/configuring-fluent-bit/multiline-parsing.md) definition. | ||
- To concatenate messages read from a log file, it's highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages originally split by Docker or CRI container engines, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support). | ||
- To concatenate messages read from a log file, it's highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages that were originally one line, but split by Docker or CRI container engines because of their size, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) icw the `docker` or `cri` parser. to concatenate application logs like stacktraces on top of that, you can use this multiline filter. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
||
{% hint style="warning" %} | ||
|
||
|
@@ -319,7 +319,8 @@ Lines that don't match a pattern aren't considered as part of the multiline mess | |
|
||
## Docker partial message use case | ||
|
||
When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16KB. If your application emits a 100K log line, it will be split into seven partial messages. If you are using the [Fluentd Docker Log Driver](https://docs.docker.com/config/containers/logging/fluentd/) to send the logs to Fluent Bit, they might look like this: | ||
When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16KB. | ||
If your application emits a 100K log line, it will be split into seven partial messages. The docker parser will merge these back to one line. If instead you are using the [Fluentd Docker Log Driver](https://docs.docker.com/config/containers/logging/fluentd/) to send the logs to Fluent Bit, they might look like this: | ||
|
||
```text | ||
{"source": "stdout", "log": "... omitted for brevity...", "partial_message": "true", "partial_id": "dc37eb08b4242c41757d4cd995d983d1cdda4589193755a22fcf47a638317da0", "partial_ordinal": "1", "partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig"}] | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need all that spacing, markdown draws the table fine without it and it's much easier to read.