-
Notifications
You must be signed in to change notification settings - Fork 474
Correct Cisco FTD Ingest Pipeline for Message IDs 302013 and 302015 (#5647) #5648
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
This seems to be a contradiction to what was assumed and discussed at: #4563 |
Hey @MakoWish thanks for submitting the PR. While we review internally to what constitutes correct assumption of |
Hi @kcreddy ,
The "inbound" and "outbound" strings in the Cisco logs are actually incorrect, and we have submit a bug report to Cisco for this issue after a brief discussion on the TAC we opened. Factually, the source will always be first interface/IP:port info in each log entry, and the destination will always be the second interface/IP:port info. Here is the complete response from the TAC we have open with Cisco:
See the two sample messages in my original post. Both are marked as "inbound", but only the first is truly an inbound request as can be seen by the interface names ("for outside... to inside..."). The second is actually an outbound request that is incorrectly marked as inbound ("for inside... to outside..."). At our request, a new bug is being submit to correct this behavior.
Yes. I will do that now. Eric |
Should a "known issue" be added to the Cisco FTD Integration noting the bug on Cisco's FTD logs? Either customers should be made aware that the
Let me know, and I can add it to the PR to fix this issue completely. Eric |
/test |
🌐 Coverage report
|
@MakoWish Thank you for digging into this, and particularly for following up with Cisco to get an answer. The test expectations need to be regenerated with For the classification based on CIDR we do have the |
I am not sure how/where I would do that, but would love to learn it if you could tell me how/where that is done.
Okay, so that does require setting the zones in the Integration's configuration to work. I don't have those set on our Integration, and I was just looking briefly at the docs and don't see any mention of it. What would be the expected values for those? That actually adds a little more justification for including some logic for inbound/outbound in the case those aren't set, and we already know the Cisco logs' interpretation of inbound/outbound is incorrect. EDIT: Okay, so the zones would be the interface names as seen in the logs, so using the example logs I posted above, the internal zone would be
That's not a bad idea. So, add an option to the manifest. Something like this?
I understand that would require modifying stream.yml.hbs:
Then the script would have to watch for
Eric |
Sorry, typo wouldn't have helped (should have beenb
From the pipeline code they would be a list of IPs (the logic depends on literal string matches). integrations/packages/cisco_ftd/data_stream/log/elasticsearch/ingest_pipeline/default.yml Lines 1858 to 1867 in 4bb210d
Yep.
Also correct.
An option would be to roll the current logic into the painless script. That woul improve the logic locality and make correctness clearer. |
Still pretty new to this whole GitHub thing... I had a little bit of time to work more on this, but my branch is saying I need to discard my 5 commits to sync with main. How can I sync without losing the commits I made? This branch has conflicts that must be resolved EDIT: Disregard. There was another merge to main that conflicted with my changes to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Incremented version number due to conflict with recent merge to main.
So, I thought a single script would get pretty ugly to handle all the logic for both zone-based or private CIDR-based network direction, so I just placed my script after the Zone-based logic with a note. Manifest has been updated with the default option of considering private networks as internal. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will also need changes in the test expectations.
# | ||
# Network Directionality when zones are not configured. | ||
# | ||
# Requires Integration option "consider_private_internal" (defaults to True) | ||
# | ||
- script: | ||
description: Calculate network.direction if zones are not configured | ||
lang: painless | ||
if: "ctx?._temp_?.consider_private_internal == true && | ||
ctx?.source?.ip != null && | ||
ctx?.destination?.ip != null && | ||
(ctx?._temp_?.external_zones == null || | ||
ctx?._temp_?.internal_zones == null)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than having non-orthogonal behaviour like this where the configurations interact with each other, I'd suggest this.
# | |
# Network Directionality when zones are not configured. | |
# | |
# Requires Integration option "consider_private_internal" (defaults to True) | |
# | |
- script: | |
description: Calculate network.direction if zones are not configured | |
lang: painless | |
if: "ctx?._temp_?.consider_private_internal == true && | |
ctx?.source?.ip != null && | |
ctx?.destination?.ip != null && | |
(ctx?._temp_?.external_zones == null || | |
ctx?._temp_?.internal_zones == null)" | |
# | |
# Network Directionality when zones do not resolve direction. | |
# | |
# Requires Integration option "consider_private_internal" (defaults to True) | |
# | |
- script: | |
description: Calculate network.direction if zones have not resolved direction. | |
lang: painless | |
if: 'ctx._temp_?.consider_private_internal == true && | |
ctx.source?.ip != null && ctx.destination?.ip != null && | |
(ctx.network?.direction == null || ["", "unknown"].contains(ctx.network.direction))' |
This means that if both are configured private ranges can be used in conjunction with the known zones, falling back to the private ranges if the known zones fail to resolve direction.
This behaviour (or the behaviour in the current proposal) needs to be described in the user-facing documentation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately, your suggested change will prevent the script from ever running.
The Grok parsing of these messages already assigned network.direction
(incorrectly based on Cisco bug), so the value will never be null
or ""
. After that, if Zones are not configured, the zone-based logic will not run, so it will also never be "unknown"
. With that conditional you added, the script will never run.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can remove the network.direction setting in the grok.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have only identified this network.direction
issue in the two Message ID's noted in this PR, as they seem to be very common Message ID's in our environment, but given the admission from Cisco, the issue would have to be present for all Message ID's that include a direction string in the message. Are you suggesting I remove the network.direction
setting in all Grok and Dissect patterns? That would be the only way your suggested change would work. I also don't know what "non-orthogonal" means, hahaha.
Pinging @efd6
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that's my suggestion.
Orthogonality of features is when changing one does not affect the other. This is generally a fairly helpful thing to avoid surprises. When I'm driving my car, I don't want the airconditioner control to interact with the brakes for example. In this particular case making a change in one setting gives surprising behaviour from another.
Should be good to go. |
/test |
Merge time? :-D |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to include at least one test for the cases that result in a direction being set, exercising the new painless code.
…Wish/integrations into cisco_ftd_ids_302013_302015
I could not for the life of me get it to work with the variable going to The script is working as expected when running the test, and I can confirm |
@MakoWish I've added configuration and some additional cases to test the script. Is this what you were concerned about? Also, looking at the tag name, I'm wondering if "private_is_internal" would be better than "consider_private_internal". The configuration would remain the same, just changing the tag. WDYT? |
/test |
Yes! That did it! I suppose I would have seen it if I looked closer at the other examples, but I was not aware you had to do individual configs for the test logs.
That one doesn't matter to me one bit. If you think that is better, I will get it changed out now. |
/test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This LGTM but I'd like to get a second review since it's sort of subtle.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The changes themself seems fine, a few questions before we can merge @MakoWish .
In the testdata that is in the integration, all the network directions are now empty for some of the older testdata, rather than reversed, is this intended? I would assume the change would still populate the direction itself?
EDIT: Just wanted to give a shoutout for pushing this, it seems like a really great job and its greatly appreciated!
@@ -38,7 +38,6 @@ | |||
} | |||
}, | |||
"network": { | |||
"direction": "outbound", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are removed rather than reverted to inbound etc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @P1llus,
We removed all Grok and Dissect parsing of network.direction
(since it won't be accurate anyway), so the only way it will be populated is by either having zones configured, or specifying this new setting of private_is_internal
.
I did not specify zones for the tests, so it was not populated with that method.
Although this new private_is_internal
defaults to true, I was not aware the only way to test the new setting was to expliciitly set it for each of the test logs with a test-*-config.yml
file. The private_is_internal
was only set by @efd6 for the new test file I created (thank you for that!), and you can see network.direction
was populated in test-ftd-inbound-outbound.log-expected.json
.
Since this new setting defaults to true, let me know if you want me to create a *-config.yml for each of the other existing test logs and regenerate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MakoWish The config in the pipeline test sets fields, while the manifest sets values in a template, so it requires to be run as a system test to be picked up. I'd rather not include the private_is_internal
for all tests (I added only to the inbound/outbound to capture it's behaviour, but seeing the behaviour when it's not set is important too).
@P1llus The rationale for not including the field from the logs was that it appears to be bugged in the Cisco output (per @MakoWish's comment in the PR description), so if we can't calculate it, it's better to have no information than wrong information.
Leaving until @P1llus has resolved this conversation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pinging @P1llus
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any more questions or issues you would like to address on this one, @P1llus ?
Had to correct some conflicts with a recent merge. Can someone kick off another test again? Then @P1llus, please let me know if there is anything else outstanding on this one to get it merged. Thank you |
/test |
Good from my side! |
required: true | ||
show_user: true | ||
title: Consider private networks as internal | ||
description: Assumes CIDR ranges `10.0.0.0/8`, `172.16.0.0/12`, and `192.168.0.0/16` are internal networks. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The note above should probably be here too, and below. I'll do this.
…lastic#5647) (elastic#5648) Disregard network direction information from Cisco logs which appears to be inconsistently rendered. Add an option to allow determination of network.direction from private network status if zone information is unable to resolve direction.
Type of PR
Bug
What does this PR do?
The Ingest Pipeline for Cisco FTD Message ID's 302013 and 302015 is incorrectly parsing the source and destination details opposite for messages Cisco has marked as "incoming". Of the three Grok patterns provided in the pipeline, the first is correct but redundant to the third, and the second is incorrect. The third pattern will handle all instances of 302013 and 302015 messages, so the first two should be removed.
EDIT: Cisco has confirmed the "inbound" or "outbound" information in these messages is inconsistent (see note in my comment below), and a bug report is being submit on our behalf.
Checklist
changelog.yml
file.How to test this PR locally
Regardless of the "inbound" or "outbound" Cisco is putting in Message ID's 302013 and 302015, the "for" details should always be the source, and the "to" details should always be the destination. The following messages may be used for testing.
Related issues
Closes #5647
Screenshots