[Kafka] New custom input package

[P1llus]

What does this PR do?

This adds custom Kafka input package, allowing users to ingest data from kafka using our input, with their own pipelines.

This has been built in the same format and principle as our other input packages.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[P1llus]

Should work just fine, but I want to see if we can add some kafka system tests, looking for some proper docker images to do this first.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-04-27T14:39:23.458+0000

• Duration: 15 min 39 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3
Skipped	0
Total	3

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[kaiyan-sheng]

Overall it looks good to me. I think we still need system tests here in order to release it as GA right?
About the owner, not sure if it's appropriate to change right now but I think eventually elastic/obs-service-integrations should own this.

[jsoriano]

About the owner, not sure if it's appropriate to change right now but I think eventually elastic/obs-service-integrations should own this.

+1 to this, I think we shouldn't have packages owned by integrations as this is becoming confusing, see #3202

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚 2.857
Classes	100.0% (0/0)	💚 2.857
Methods	100.0% (3/3)	💚 11.905
Lines	100.0% (0/0)	💚 11.162
Conditionals	100.0% (0/0)	💚

[lalit-satapathy]

Just trying to clarify. This new package is developed by security team but not owned by them?

[P1llus]

/test

[lalit-satapathy]

@P1llus the changes looks good to me. The name "custom kafka logs" confused me, where I thought as the logs of kafka broker. Where as this is actually just referring to all the messages in "kafka topic".