Skip to content

Correct Cisco FTD Ingest Pipeline for Message IDs 302013 and 302015 (#5647) #5648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
May 31, 2023
Merged

Correct Cisco FTD Ingest Pipeline for Message IDs 302013 and 302015 (#5647) #5648

merged 30 commits into from
May 31, 2023

Conversation

MakoWish
Copy link
Contributor

@MakoWish MakoWish commented Mar 22, 2023

Type of PR

Bug

What does this PR do?

The Ingest Pipeline for Cisco FTD Message ID's 302013 and 302015 is incorrectly parsing the source and destination details opposite for messages Cisco has marked as "incoming". Of the three Grok patterns provided in the pipeline, the first is correct but redundant to the third, and the second is incorrect. The third pattern will handle all instances of 302013 and 302015 messages, so the first two should be removed.

EDIT: Cisco has confirmed the "inbound" or "outbound" information in these messages is inconsistent (see note in my comment below), and a bug report is being submit on our behalf.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Regardless of the "inbound" or "outbound" Cisco is putting in Message ID's 302013 and 302015, the "for" details should always be the source, and the "to" details should always be the destination. The following messages may be used for testing.

<166>Mar 22 2023 01:44:32 fw-contoso-1 : %FTD-6-302013: Built inbound TCP connection 123456789 for IF-Outside:190.97.114.166/19884 (190.97.114.166/19884) to IF-Inside:192.168.0.10/22 (8.8.8.8/22)
<166>Mar 22 2023 17:30:25 fw-contoso-1 : %FTD-6-302013: Built inbound TCP connection 234567891 for IF-Inside:192.168.0.20/61863 (8.8.8.8/16013) to IF-Outside:104.18.32.68/443(104.18.32.68/443)

Related issues

Closes #5647

Screenshots

@MakoWish MakoWish requested review from a team as code owners March 22, 2023 21:10
@elasticmachine
Copy link

elasticmachine commented Mar 22, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-05-31T06:07:29.224+0000

  • Duration: 17 min 57 sec

Test stats 🧪

Test Results
Failed 0
Passed 22
Skipped 0
Total 22

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@MakoWish MakoWish changed the title Correct Cisco FTD Ingest Pipeline Incorrect for Message IDs 302013 and 302015 (#5647) Correct Cisco FTD Ingest Pipeline for Message IDs 302013 and 302015 (#5647) Mar 22, 2023
@ari-aviran ari-aviran removed the request for review from a team March 23, 2023 07:25
@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@kcreddy
Copy link
Contributor

kcreddy commented Mar 25, 2023

This seems to be a contradiction to what was assumed and discussed at: #4563
Seeking more clarity internally.

@kcreddy
Copy link
Contributor

kcreddy commented Mar 25, 2023

Hey @MakoWish thanks for submitting the PR. While we review internally to what constitutes correct assumption of source and destination, could you add entry to changelog.yml and also update its corresponding version in manifest.yml?

@MakoWish
Copy link
Contributor Author

MakoWish commented Mar 25, 2023

Hi @kcreddy ,

This seems to be a contradiction to what was assumed and discussed at: #4563 Seeking more clarity internally.

The "inbound" and "outbound" strings in the Cisco logs are actually incorrect, and we have submit a bug report to Cisco for this issue after a brief discussion on the TAC we opened. Factually, the source will always be first interface/IP:port info in each log entry, and the destination will always be the second interface/IP:port info. Here is the complete response from the TAC we have open with Cisco:

On the ASAs the inbound/outbound direction was determined based on the security levels assigned in the interfaces (this was not defined by the logical name or by the IP addresses).
lower to higher = inbound
higher to lower = outbound

However, on the FTDs, all security levels are set to 0, and those cannot be changed. Also, there is no external or internal documentation describing a new criterion to define the inbound/outbound string on those syslog messages. Because of this, that inbound/outbound tag is inconsistent on the FTD syslogs.

If you need a solution for this I will need to engage further resources in this case. Most likely we will end with a new bug or enhancement request to modify this behavior in future releases.

See the two sample messages in my original post. Both are marked as "inbound", but only the first is truly an inbound request as can be seen by the interface names ("for outside... to inside..."). The second is actually an outbound request that is incorrectly marked as inbound ("for inside... to outside...").

At our request, a new bug is being submit to correct this behavior.

could you add entry to changelog.yml and also update its corresponding version in manifest.yml?

Yes. I will do that now.

Eric

@MakoWish
Copy link
Contributor Author

MakoWish commented Mar 25, 2023

Should a "known issue" be added to the Cisco FTD Integration noting the bug on Cisco's FTD logs? Either customers should be made aware that the network.direction will not be accurate (as confirmed by Cisco), or perhaps we should add some logic in the pipeline to override it? Something along the lines of:

  - script:
      description: Calculate network.direction due to bug in Cisco FTD logs
      lang: painless
      if: "ctx?.source?.ip != null && ctx?.destination?.ip != null"
      source: |
        boolean isPrivateCIDR(def ip) {
          CIDR class_a_network = new CIDR('10.0.0.0/8');
          CIDR class_b_network = new CIDR('172.16.0.0/12');
          CIDR class_c_network = new CIDR('192.168.0.0/16');

          try {
            return class_a_network.contains(ip) || class_b_network.contains(ip) || class_c_network.contains(ip);
          } catch (IllegalArgumentException e) {
            return false;
          }
        }
        try {
          if (ctx.network == null) {
            Map map = new HashMap();
            ctx.put('network', map);
          }

          if (!isPrivateCIDR(ctx.source.ip) && isPrivateCIDR(ctx.destination.ip)) {
            ctx.network.direction = 'inbound';
          } else if (isPrivateCIDR(ctx.source.ip) && !isPrivateCIDR(ctx.destination.ip)) {
            ctx.network.direction = 'outbound';
          } else if (isPrivateCIDR(ctx.source.ip) && isPrivateCIDR(ctx.destination.ip)) {
            ctx.network.direction = 'internal';
          } else if (!isPrivateCIDR(ctx.source.ip) && !isPrivateCIDR(ctx.destination.ip)) {
            ctx.network.direction = 'external';
          } else {
            ctx.network.direction = 'unknown';
          }
        }
        catch (Exception e) {
          ctx.network.direction = null;
        }

Let me know, and I can add it to the PR to fix this issue completely.

Eric

@efd6
Copy link
Contributor

efd6 commented Mar 26, 2023

/test

@elasticmachine
Copy link

elasticmachine commented Mar 26, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (1/1) 💚
Files 100.0% (1/1) 💚
Classes 100.0% (1/1) 💚
Methods 100.0% (22/22) 💚
Lines 70.573% (1403/1988)
Conditionals 100.0% (0/0) 💚

@efd6
Copy link
Contributor

efd6 commented Mar 26, 2023

@MakoWish Thank you for digging into this, and particularly for following up with Cisco to get an answer.

The test expectations need to be regenerated with elastic package test pipeline -g. Are you able to do this?

For the classification based on CIDR we do have the internal_zones and external_zones (here) which could be used to do similar work. The script would been to not collide with that logic and if it is added I would like to see a note added to the manifest explaining that private networks are considered internal (or alternatively an option defaulting to true to consider private networks as internal).

@MakoWish
Copy link
Contributor Author

MakoWish commented Mar 28, 2023

The test expectations need to be regenerated with elastic package test pipeline -g. Are you able to do this?

I am not sure how/where I would do that, but would love to learn it if you could tell me how/where that is done.

For the classification based on CIDR we do have the internal_zones and external_zones (here) which could be used to do similar work.

Okay, so that does require setting the zones in the Integration's configuration to work. I don't have those set on our Integration, and I was just looking briefly at the docs and don't see any mention of it. What would be the expected values for those? That actually adds a little more justification for including some logic for inbound/outbound in the case those aren't set, and we already know the Cisco logs' interpretation of inbound/outbound is incorrect. EDIT: Okay, so the zones would be the interface names as seen in the logs, so using the example logs I posted above, the internal zone would be IF-Internal, and the external zone would be IF-External. Got that!

or alternatively an option defaulting to true to consider private networks as internal

That's not a bad idea. So, add an option to the manifest. Something like this?

      - name: consider_private_internal
        required: true
        show_user: true
        title: Consider private networks as internal
        description: Assumes CIDR ranges `10.0.0.0/8`, `172.16.0.0/12`, and `192.168.0.0/16` are internal networks.
        type: bool
        multi: false
        default: true

I understand that would require modifying stream.yml.hbs:

{{#if consider_private_internal.length}}
- add_fields:
    target: _temp_
    fields:
      consider_private_internal:
        - true
{{/if}}

Then the script would have to watch for _temp_.consider_private_internal, and to not interfere with the zone logic, also include that. Basically, if we are to consider private networks as internal, we have both a source and destination IP, but we don't have one or both of the zones configured:

  - script:
      description: Calculate network.direction if zones are not configured
      lang: painless
      if: "ctx?._temp_?.consider_private_internal == true &&
           ctx?.source?.ip != null && 
           ctx?.destination?.ip != null &&
          (ctx?._temp_?.external_zones == null ||
           ctx?._temp_?.internal_zones == null)"
      ...

Eric

@efd6
Copy link
Contributor

efd6 commented Mar 28, 2023

Sorry, typo wouldn't have helped (should have beenb elastic-package test pipeline -g), but here are the docs.

What would be the expected values for those?

From the pipeline code they would be a list of IPs (the logic depends on literal string matches).

- set:
field: network.direction
value: inbound
if: >
ctx?._temp_?.external_zones != null &&
ctx?._temp_?.internal_zones != null &&
ctx?.observer?.ingress?.zone != null &&
ctx?.observer?.egress?.zone != null &&
ctx._temp_.external_zones.contains(ctx.observer.ingress.zone) &&
ctx._temp_.internal_zones.contains(ctx.observer.egress.zone)

So, add an option to the manifest. Something like this?

   ...code...

Yep.

I understand that would require modifying stream.yml.hbs:

Also correct.

Then the script would have to watch for temp.consider_private_internal, and to not interfere with the zone logic, also include that. Basically, if we are to consider private networks as internal, we have both a source and destination IP, but we don't have one or both of the zones configured:

An option would be to roll the current logic into the painless script. That woul improve the logic locality and make correctness clearer.

@MakoWish
Copy link
Contributor Author

MakoWish commented Mar 31, 2023

Still pretty new to this whole GitHub thing... I had a little bit of time to work more on this, but my branch is saying I need to discard my 5 commits to sync with main. How can I sync without losing the commits I made?

This branch has conflicts that must be resolved
Discard 5 commits to make this branch match the upstream repository. 5 commits will be removed from this branch.
You can resolve merge conflicts using the command line and a text editor.

EDIT: Disregard. There was another merge to main that conflicted with my changes to changelog.yml and manifest.yml. Manually corrected those, and the warning is gone.

Copy link
Contributor Author

@MakoWish MakoWish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremented version number due to conflict with recent merge to main.

@MakoWish
Copy link
Contributor Author

So, I thought a single script would get pretty ugly to handle all the logic for both zone-based or private CIDR-based network direction, so I just placed my script after the Zone-based logic with a note. Manifest has been updated with the default option of considering private networks as internal.

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will also need changes in the test expectations.

Comment on lines 1922 to 1934
#
# Network Directionality when zones are not configured.
#
# Requires Integration option "consider_private_internal" (defaults to True)
#
- script:
description: Calculate network.direction if zones are not configured
lang: painless
if: "ctx?._temp_?.consider_private_internal == true &&
ctx?.source?.ip != null &&
ctx?.destination?.ip != null &&
(ctx?._temp_?.external_zones == null ||
ctx?._temp_?.internal_zones == null)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than having non-orthogonal behaviour like this where the configurations interact with each other, I'd suggest this.

Suggested change
#
# Network Directionality when zones are not configured.
#
# Requires Integration option "consider_private_internal" (defaults to True)
#
- script:
description: Calculate network.direction if zones are not configured
lang: painless
if: "ctx?._temp_?.consider_private_internal == true &&
ctx?.source?.ip != null &&
ctx?.destination?.ip != null &&
(ctx?._temp_?.external_zones == null ||
ctx?._temp_?.internal_zones == null)"
#
# Network Directionality when zones do not resolve direction.
#
# Requires Integration option "consider_private_internal" (defaults to True)
#
- script:
description: Calculate network.direction if zones have not resolved direction.
lang: painless
if: 'ctx._temp_?.consider_private_internal == true &&
ctx.source?.ip != null && ctx.destination?.ip != null &&
(ctx.network?.direction == null || ["", "unknown"].contains(ctx.network.direction))'

This means that if both are configured private ranges can be used in conjunction with the known zones, falling back to the private ranges if the known zones fail to resolve direction.

This behaviour (or the behaviour in the current proposal) needs to be described in the user-facing documentation.

Copy link
Contributor Author

@MakoWish MakoWish Apr 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, your suggested change will prevent the script from ever running.

The Grok parsing of these messages already assigned network.direction (incorrectly based on Cisco bug), so the value will never be null or "". After that, if Zones are not configured, the zone-based logic will not run, so it will also never be "unknown". With that conditional you added, the script will never run.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove the network.direction setting in the grok.

Copy link
Contributor Author

@MakoWish MakoWish Apr 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have only identified this network.direction issue in the two Message ID's noted in this PR, as they seem to be very common Message ID's in our environment, but given the admission from Cisco, the issue would have to be present for all Message ID's that include a direction string in the message. Are you suggesting I remove the network.direction setting in all Grok and Dissect patterns? That would be the only way your suggested change would work. I also don't know what "non-orthogonal" means, hahaha.

Pinging @efd6

Copy link
Contributor

@efd6 efd6 Apr 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's my suggestion.

Orthogonality of features is when changing one does not affect the other. This is generally a fairly helpful thing to avoid surprises. When I'm driving my car, I don't want the airconditioner control to interact with the brakes for example. In this particular case making a change in one setting gives surprising behaviour from another.

@MakoWish
Copy link
Contributor Author

Should be good to go.

@efd6
Copy link
Contributor

efd6 commented May 1, 2023

/test

@MakoWish
Copy link
Contributor Author

MakoWish commented May 8, 2023

Merge time? :-D

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to include at least one test for the cases that result in a direction being set, exercising the new painless code.

@MakoWish
Copy link
Contributor Author

MakoWish commented May 9, 2023

I could not for the life of me get it to work with the variable going to _temp_, so I put it in the tags like preserve_original_event is, and it worked fine.

The script is working as expected when running the test, and I can confirm network.direction is properly being set. However, no matter what I try, I cannot get the expected results document to get the network.direction to work. Is there a different method of ensuring that setting is applied to the expected results?

@efd6
Copy link
Contributor

efd6 commented May 10, 2023

@MakoWish I've added configuration and some additional cases to test the script. Is this what you were concerned about?

Also, looking at the tag name, I'm wondering if "private_is_internal" would be better than "consider_private_internal". The configuration would remain the same, just changing the tag. WDYT?

@efd6
Copy link
Contributor

efd6 commented May 10, 2023

/test

@MakoWish
Copy link
Contributor Author

@MakoWish I've added configuration and some additional cases to test the script. Is this what you were concerned about?

Yes! That did it! I suppose I would have seen it if I looked closer at the other examples, but I was not aware you had to do individual configs for the test logs.

Also, looking at the tag name, I'm wondering if "private_is_internal" would be better than "consider_private_internal". The configuration would remain the same, just changing the tag. WDYT?

That one doesn't matter to me one bit. If you think that is better, I will get it changed out now.

@efd6
Copy link
Contributor

efd6 commented May 11, 2023

/test

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM but I'd like to get a second review since it's sort of subtle.

Copy link
Member

@P1llus P1llus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes themself seems fine, a few questions before we can merge @MakoWish .
In the testdata that is in the integration, all the network directions are now empty for some of the older testdata, rather than reversed, is this intended? I would assume the change would still populate the direction itself?

EDIT: Just wanted to give a shoutout for pushing this, it seems like a really great job and its greatly appreciated!

@@ -38,7 +38,6 @@
}
},
"network": {
"direction": "outbound",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are removed rather than reverted to inbound etc?

Copy link
Contributor Author

@MakoWish MakoWish May 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @P1llus,

We removed all Grok and Dissect parsing of network.direction (since it won't be accurate anyway), so the only way it will be populated is by either having zones configured, or specifying this new setting of private_is_internal.

I did not specify zones for the tests, so it was not populated with that method.

Although this new private_is_internal defaults to true, I was not aware the only way to test the new setting was to expliciitly set it for each of the test logs with a test-*-config.yml file. The private_is_internal was only set by @efd6 for the new test file I created (thank you for that!), and you can see network.direction was populated in test-ftd-inbound-outbound.log-expected.json.

Since this new setting defaults to true, let me know if you want me to create a *-config.yml for each of the other existing test logs and regenerate.

Copy link
Contributor

@efd6 efd6 May 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MakoWish The config in the pipeline test sets fields, while the manifest sets values in a template, so it requires to be run as a system test to be picked up. I'd rather not include the private_is_internal for all tests (I added only to the inbound/outbound to capture it's behaviour, but seeing the behaviour when it's not set is important too).

@P1llus The rationale for not including the field from the logs was that it appears to be bugged in the Cisco output (per @MakoWish's comment in the PR description), so if we can't calculate it, it's better to have no information than wrong information.

Leaving until @P1llus has resolved this conversation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinging @P1llus

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any more questions or issues you would like to address on this one, @P1llus ?

@MakoWish
Copy link
Contributor Author

Had to correct some conflicts with a recent merge.

Can someone kick off another test again? Then @P1llus, please let me know if there is anything else outstanding on this one to get it merged.

Thank you

@efd6
Copy link
Contributor

efd6 commented May 30, 2023

/test

@P1llus
Copy link
Member

P1llus commented May 31, 2023

Good from my side!

required: true
show_user: true
title: Consider private networks as internal
description: Assumes CIDR ranges `10.0.0.0/8`, `172.16.0.0/12`, and `192.168.0.0/16` are internal networks.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The note above should probably be here too, and below. I'll do this.

@efd6 efd6 merged commit ff5d6b8 into elastic:main May 31, 2023
@MakoWish MakoWish deleted the cisco_ftd_ids_302013_302015 branch May 31, 2023 13:06
agithomas pushed a commit to agithomas/integrations that referenced this pull request Jun 5, 2023
…lastic#5647) (elastic#5648)

Disregard network direction information from Cisco logs which appears to
be inconsistently rendered.

Add an option to allow determination of network.direction from private
network status if zone information is unable to resolve direction.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug in Cisco FTD Integration's Ingest Pipeline for Message ID's 302013, 302015
6 participants