[Cisco IOS] Fix parsing when message_count is missing #6919

chemamartinez · 2023-07-11T15:45:41Z

What does this PR do?

This pull request fixes a parsing error in the Cisco IOS integration when parsing an event with the format <priority>TIMESTAMP|HOSTNAME.

Currently, the grok processor expects for the message_count field, which is optional so it could not come in the event.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Regarding this sample log:

<189>Jun 28 17:52:49 192.168.100.2 sw01: Jun 28 10:52:48.876 PST: %SYS-5-CONFIG_I: Configured from console by akroh on vty0 (10.100.11.10)

It starts by a priority label but doesn't contain a message count before the timestamp and hostname. For this case, currently the ingest pipeline fails.

# elastic-package test pipeline --generate
Run pipeline tests for the package
--- Test results for package: cisco_ios - START ---
FAILURE DETAILS:
cisco_ios/log test-syslog.log:
[0] unexpected pipeline error: [Processor "grok" with tag "grok_header" in pipeline "default-1689088307236906000" failed with message "Provided Grok expressions do not match field value: [<189>Jun 28 17:52:49 192.168.100.2 sw01: Jun 28 10:52:48.876 PST:]"]
[1] unexpected pipeline error: [Processor "grok" with tag "grok_header" in pipeline "default-1689088307236906000" failed with message "Provided Grok expressions do not match field value: [<189>Jun 28 17:52:49.964:]"]
[2] unexpected pipeline error: [Processor "grok" with tag "grok_header" in pipeline "default-1689088307236906000" failed with message "Provided Grok expressions do not match field value: [<189>sw01: Jan  6 2022 21:01:34.964:]"]


╭───────────┬─────────────┬───────────┬───────────────────────────────┬─────────────────────────────────────────────────────────────────────────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                     │ RESULT                                                                      │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼───────────────────────────────┼─────────────────────────────────────────────────────────────────────────────┼──────────────┤
│ cisco_ios │ log         │ pipeline  │ test-asr920.log               │ PASS                                                                        │    2.17625ms │
│ cisco_ios │ log         │ pipeline  │ test-cisco-ios.log            │ PASS                                                                        │   3.948209ms │
│ cisco_ios │ log         │ pipeline  │ test-date-format-tzoffset.log │ PASS                                                                        │   1.910292ms │
│ cisco_ios │ log         │ pipeline  │ test-date-format.log          │ PASS                                                                        │   2.490458ms │
│ cisco_ios │ log         │ pipeline  │ test-syslog.log               │ FAIL: test case failed: one or more problems with fields found in documents │   2.298916ms │
╰───────────┴─────────────┴───────────┴───────────────────────────────┴─────────────────────────────────────────────────────────────────────────────┴──────────────╯
--- Test results for package: cisco_ios - END   ---
Done
Error: one or more test cases failed

After the new pattern, the ingest pipeline is able to parse it properly.

# elastic-package test pipeline --generate
Run pipeline tests for the package
--- Test results for package: cisco_ios - START ---
╭───────────┬─────────────┬───────────┬───────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                     │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼───────────────────────────────┼────────┼──────────────┤
│ cisco_ios │ log         │ pipeline  │ test-asr920.log               │ PASS   │   2.203667ms │
│ cisco_ios │ log         │ pipeline  │ test-cisco-ios.log            │ PASS   │   3.858792ms │
│ cisco_ios │ log         │ pipeline  │ test-date-format-tzoffset.log │ PASS   │   1.815875ms │
│ cisco_ios │ log         │ pipeline  │ test-date-format.log          │ PASS   │    2.12875ms │
│ cisco_ios │ log         │ pipeline  │ test-syslog.log               │ PASS   │   1.512334ms │
╰───────────┴─────────────┴───────────┴───────────────────────────────┴────────┴──────────────╯
--- Test results for package: cisco_ios - END   ---
Done

elasticmachine · 2023-07-11T15:48:40Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2023-07-11T15:57:29Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-24T06:29:54.197+0000
Duration: 18 min 7 sec

Test stats 🧪

Test	Results
Failed	0
Passed	13
Skipped	0
Total	13

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-07-11T16:05:24Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚 2.861
Classes	100.0% (`1/1`)	💚 2.861
Methods	100.0% (`15/15`)	💚 7.777
Lines	91.193% (`321/352`)	👍 0.07
Conditionals	100.0% (`0/0`)	💚

andrewkroh · 2023-07-11T19:15:54Z

I have another format that I don't think is handled. It's coming from Cisco IOS Software, C3750E Software (C3750E-IPBASEK9-M), Version 15.0(2)SE11, RELEASE SOFTWARE (fc3). Let me know if you think I should open a separate issue.

{
  "event.original": [
    "<190>3132517: 3132513: Jul 11 2023 18:38:06.203 UTC: %SEC-6-IPACCESSLOGP: list 110 denied tcp 10.100.18.5(53222) -> 10.100.16.35(7), 1 packet"
  ]
}

From https://www.cisco.com/c/en/us/td/docs/routers/ios/config/17-x/syst-mgmt/b-system-management/m_esm-syslog.html:

When messages are received on a syslog host a “syslog-count” number is also added:
<syslog-count>: <sequence-number>: <time stamp>:%<facility>-<severity>-<mnemonic>: <message-text>

Related cisco config:

service timestamps log datetime msec year
service sequence-numbers

logging message-counter syslog is not in the config so this must be enabled by default.

andrewkroh · 2023-07-11T19:37:26Z

To get some more log diversity I enabled a few different origin ID options (the earlier sample was with no logging origin-id).

logging origin-id ipv6

<190>3132783: ::: 3132779: Jul 11 2023 19:25:44.568 UTC: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 96 packets

logging origin-id ip

<189>3132785: 0.0.0.0: 3132781: Jul 11 2023 19:30:04.771 UTC: %SYS-5-CONFIG_I: Configured from console by akroh on vty0 (10.100.11.10)

logging origin-id hostname

<190>3132811: sw01: 3132807: Jul 11 2023 19:30:44.575 UTC: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 96 packets

logging origin-id string custom.string
<189>3132813: custom.string: 3132809: Jul 11 2023 19:32:08.116 UTC: %SYS-5-CONFIG_I: Configured from console by akroh on vty0 (10.100.11.10)

chemamartinez · 2023-07-13T11:35:28Z

I've tested them with this current change and all of them are ingested with no error, but some fields are not mapped well. I can extend the pattern of the grok processor to consider all these fields.

I have just one concern with the last example

logging origin-id string custom.string
<189>3132813: custom.string: 3132809: Jul 11 2023 19:32:08.116 UTC: %SYS-5-CONFIG_I: Configured from console by akroh on vty0 (10.100.11.10)

about how to differentiate a custom string with a hostname.

chemamartinez · 2023-07-13T19:04:21Z

@andrewkroh,

I've added a new commit (ac5d0254c826180031ea541c5a6672916b41a9b1) trying to cover all the possibilities in the syslog header. However, I am still struggling with some use cases.

There are some logs where the hostname is missing. In these cases, the sequence-number is taken as the hostname since it is located in its place and a number is also a valid hostname.
Including a custom string is also tricky to take into account since it is also parsed as a hostname.

Apart from that, I think some new use cases are now handled, such as the parse of the syslog's timestamp and IP, or the possibility of having an asterisk before the timestamp when it may not be synchronized, as the documentation explains.

andrewkroh

Including a custom string is also tricky to take into account since it is also parsed as a hostname.

I would treat the custom string value as a hostname. I think that its general purpose is the same, to identify the host.

packages/cisco_ios/data_stream/log/fields/fields.yml

packages/cisco_ios/data_stream/log/_dev/test/pipeline/test-syslog-header.log-expected.json

packages/cisco_ios/data_stream/log/elasticsearch/ingest_pipeline/default.yml

chemamartinez · 2023-07-18T15:42:44Z

Hi @andrewkroh,

I have already submitted all the changes we have talked about.

andrewkroh

Other than the error.message suggestion, LGTM.

andrewkroh · 2023-07-20T15:02:22Z

packages/cisco_ios/data_stream/log/elasticsearch/ingest_pipeline/default.yml

+      on_failure:
+        - append:
+            field: error.message
+            value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag fail-{{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'


Why is the real tag name prefixed with "fail-"? That seems misleading. If a user searches for that tag in the pipeline or the GET /_nodes/stats?metric=ingest&filter_path=nodes.*.ingest.pipelines ingest stats data they won't find it.

This is a design scar. I'll go through the a packages and remove them so we don't have a continuing source of precedent.

Thanks for letting me know, I didn't realize that detail.

elasticmachine · 2023-07-24T07:24:40Z

Package cisco_ios - 1.16.2 containing this change is available at https://epr.elastic.co/search?package=cisco_ios

chemamartinez added Integration:cisco_ios Cisco IOS bugfix Pull request that fixes a bug issue labels Jul 11, 2023

chemamartinez self-assigned this Jul 11, 2023

chemamartinez added the Team:Security-External Integrations label Jul 11, 2023

chemamartinez marked this pull request as ready for review July 11, 2023 15:48

chemamartinez requested a review from a team as a code owner July 11, 2023 15:48

LaZyDK mentioned this pull request Jul 12, 2023

[cisco_ios] Fix grok to be aware of fman_fp_image #6930

Merged

4 tasks

chemamartinez force-pushed the sdh-3594-fix-cisco_ios-parsing branch from 5d02b23 to ac5d025 Compare July 13, 2023 19:02

chemamartinez force-pushed the sdh-3594-fix-cisco_ios-parsing branch from ac5d025 to 3bf0b3d Compare July 14, 2023 07:17

andrewkroh reviewed Jul 17, 2023

View reviewed changes

chemamartinez force-pushed the sdh-3594-fix-cisco_ios-parsing branch from b1ba87a to e4e1649 Compare July 18, 2023 11:56

andrewkroh reviewed Jul 18, 2023

View reviewed changes

packages/cisco_ios/data_stream/log/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

chemamartinez requested a review from andrewkroh July 18, 2023 15:42

chemamartinez added 7 commits July 19, 2023 08:48

Fix ingest_pipeline to support missing message_count

ff2cb8c

Add sample logs to pipeline tests

6b41a4e

Update changelog

8431a37

Fix grok processor to parse syslog header

237db74

Update docs

ce6cb96

Fix the parse of hostname and sequence in syslog header

9a789e2

Handle on failure situation in grok header

6265ffc

chemamartinez force-pushed the sdh-3594-fix-cisco_ios-parsing branch from 0859cf5 to 6265ffc Compare July 19, 2023 06:48

andrewkroh approved these changes Jul 20, 2023

View reviewed changes

efd6 mentioned this pull request Jul 24, 2023

all: remove confusing fail- prefix from error messages #7105

Merged

4 tasks

Fix error message value in grok processor

dc2911a

chemamartinez merged commit d3e0b9a into elastic:main Jul 24, 2023

chemamartinez deleted the sdh-3594-fix-cisco_ios-parsing branch February 6, 2025 10:30

[Cisco IOS] Fix parsing when message_count is missing #6919

[Cisco IOS] Fix parsing when message_count is missing #6919

Uh oh!

Conversation

chemamartinez commented Jul 11, 2023

What does this PR do?

Checklist

How to test this PR locally

Uh oh!

elasticmachine commented Jul 11, 2023

Uh oh!

elasticmachine commented Jul 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Jul 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

andrewkroh commented Jul 11, 2023

Uh oh!

andrewkroh commented Jul 11, 2023

Uh oh!

chemamartinez commented Jul 13, 2023

Uh oh!

chemamartinez commented Jul 13, 2023

Uh oh!

andrewkroh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chemamartinez commented Jul 18, 2023

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

andrewkroh Jul 20, 2023

Choose a reason for hiding this comment

Uh oh!

efd6 Jul 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chemamartinez Jul 24, 2023

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 24, 2023

Uh oh!

Uh oh!

elasticmachine commented Jul 11, 2023 •

edited

Loading

elasticmachine commented Jul 11, 2023 •

edited

Loading

andrewkroh left a comment •

edited

Loading

efd6 Jul 23, 2023 •

edited

Loading