Fix adding processors in cloudfront logs

[kaiyan-sheng]

What does this PR do?

This PR is to fix adding processors for cloudfront logs in AWS package. With the change, we can see processors in agent policy is in the right format now:

inputs:
  - id: aws-s3-cloudfront-096288f1-4c9d-4f1b-a6b1-7e73ecc7a849
    name: aws-1
    revision: 1
    type: aws-s3
    use_output: default
    meta:
      package:
        name: aws
        version: 1.24.3
    data_stream:
      namespace: default
    streams:
      - id: aws-s3-aws.cloudfront_logs-096288f1-4c9d-4f1b-a6b1-7e73ecc7a849
        data_stream:
          dataset: aws.cloudfront_logs
          type: logs
        queue_url: 'https://sqs.us-east-1.amazonaws.com/123/test'
        max_number_of_messages: 5
        access_key_id: a
        secret_access_key: b
        session_token: c
        tags:
          - forwarded
          - aws-cloudfront
        publisher_pipeline.disable_host: true
        processors:
          - drop_event:
              when:
                regexp:
                  message: ^#.*
          - drop_event:
              when:
                not:
                  regexp:
                    message: .*text/html.*

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Closes [AWS] cloudfront log integration failed to add processors #4394

[elasticmachine]

🚀 Benchmarks report

Package aws 👍(7) 💚(4) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
cloudwatch_logs	500000	333333.33	-166666.67 (-33.33%)	💔
route53_resolver_logs	6211.18	4587.16	-1624.02 (-26.15%)	💔

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-10-05T20:25:13.172+0000

• Duration: 33 min 16 sec

Test stats 🧪

Test	Results
Failed	0
Passed	162
Skipped	2
Total	164

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (13/13)	💚
Files	92.857% (13/14)	👎 -4.635
Classes	92.857% (13/14)	👎 -4.635
Methods	84.232% (203/241)	👎 -6.057
Lines	95.697% (5204/5438)	👍 4.062
Conditionals	100.0% (0/0)	💚

[tommyers-elastic]

thanks!

it's not ideal that indentation errors can affect behavior so severely! i guess the same is true of this type of YAML config in general. 😬

[kaiyan-sheng]

@tommyers-elastic I think the main issue is missing - in front of {{processors}} :) Thanks for the review!!

[elasticsatch]

Thanks for the fix. So this change requires input NOT to have dash in front of each processor? Could you show me some examples (or maybe a test case?)

[kaiyan-sheng]

@elasticsatch You mean requires/or not to have a dash in front of each processor on the kibana UI? You can see an example in the screenshot for the Kibana UI when adding a processor here: #4394

[elasticsatch]

Oh thanks, I overlooked the screenshot. So if I want multiple processors we should put something like this, right?

dissect:
  ...

drop_event:
  ..

And we can put blank lines in between, correct?

[kaiyan-sheng]

@elasticsatch OMG good call...... Nope right now it only support adding one processor! I will look into it tomorrow and fix it for multiple processors. Thanks!

[elasticsatch]

Good to hear and no pressure. Please take your time.

I also suggest to put the instruction somewhere in a document so users can see how to write the processors correctly. There is no good ones as of this moment so it took a little bit of time to figure it out on my own.