Opencanary #13970

colin-stubbs · 2025-05-22T13:45:21Z

Bug
Enhancement

Proposed commit message

Resolves issues #12911, #13024, #13025. Relevant to resolution of #2518.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Resolves multiple issues:

Testing:

elastic-package lint && check && build
elastic-package test system --generate
elastic-package test pipeline --generate
elastic-package test
Manual deploy on local elastic-package managed stack and ingest of logfile
Manual deploy on remote Elastic Cloud stack and ingest of webhooks from real opencanary honeypots

How to test this PR locally

Install and operate OpenCanary, ideally via docker.
Use Elastic Agent to ingest OpenCanary log file OR webhooks.
Scan OpenCanary with nmap with scripting to trigger events, e.g. nmap -sC 127.0.0.1
Review

Related issues

Screenshots

New basic summary dashboard added,

rebuild test event set, fix handling for certain error type events, add GeoIP enrichment for source.ip and destination.ip, rebuild/retest/regen sample events etc, retest dashboard

colin-stubbs · 2025-05-22T13:45:59Z

@navnit-elastic - new PR opened.

elasticmachine · 2025-05-22T16:00:47Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

navnit-elastic · 2025-05-26T12:34:46Z

This is a copy of PR #13026, which was reviewed but eventually closed due to inactivity.

kcreddy · 2025-05-26T15:54:24Z

/test

efd6 · 2025-05-26T20:48:10Z

The failure reported by CI Error: can't install the package: could not zip-install package; API status code = 422; response body = {"statusCode":422,"error":"Unprocessable Entity","message":"Document \"opencanary-96cfb6c4-bc46-4fd4-9476-e8e9550442d9\" belongs to a more recent version of Kibana [10.2.0] when the last known version is [8.9.0]."} is repeatable locally when running on the stack version specified in the manifest (v8.13.0).

I have tried running on v8.15.0 and v8.18.0. Both of these succeed, so either the Event Summary dashboard needs to be made to conform to v8.13's expectation, or the kibana version needs to be bumped in the manifest.

navnit-elastic · 2025-05-27T06:03:35Z

Thanks @efd6 for reporting, @colin-stubbs since there are major changes into the integration, I think it would be better to bump Kibana version to ^v8.18.0 in the manifest. Could you please do that? Thank you!

colin-stubbs · 2025-05-27T12:10:31Z

8.17.0 makes more sense.

@efd6 - out of curiosity what's the situation that has you running tests on an 8.13.x stack? Did you explicitly look at the constraint and fire up an 8.13.x version to test with because it had 8.13.0?

UPDATE: I realised you mentioned the CI pipeline... so I suppose it's smart enough to look at the constraint and fire up tests on the version/s listed there?

I'm typically testing with the latest elastic-package release whatever container image versions it pulls, at the moment that's 8.17.3, as well as an EC stack running 8.18.x, hence I've missed any issue with the constraint. I've not come across this kind of issue before either so I hadn't even thought to update the constraint version.

It strikes me that perhaps elastic-package should be a little more version aware and, if not have something added to it to check kibana asset versioning against any kibana constraint, at least present a warning to people that testing is occurring with a version greater than the lowest advertised compatible version, and hence version drift is potentially significant and that they should double check the constraints and assets will still work with the advertised constraint version?

I've updated constraints now to 8.17.0 after explicitly re-testing using an 8.17.0 stack.

user@box opencanary % elastic-package test      
Run asset tests for the package
2025/05/27 22:04:50  INFO License text found in "/SRC/GitHub/routedlogic/integrations/LICENSE.txt" will be included in package
--- Test results for package: opencanary - START ---
╭────────────┬─────────────┬───────────┬─────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                                                           │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼─────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ opencanary │             │ asset     │ dashboard opencanary-96cfb6c4-bc46-4fd4-9476-e8e9550442d9 is loaded │ PASS   │        792ns │
│ opencanary │ events      │ asset     │ index_template logs-opencanary.events is loaded                     │ PASS   │        209ns │
│ opencanary │ events      │ asset     │ ingest_pipeline logs-opencanary.events-0.5.0 is loaded              │ PASS   │        292ns │
╰────────────┴─────────────┴───────────┴─────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: opencanary - END   ---
Done
Run pipeline tests for the package
--- Test results for package: opencanary - START ---
╭────────────┬─────────────┬───────────┬───────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                                     │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼───────────────────────────────────────────────┼────────┼──────────────┤
│ opencanary │ events      │ pipeline  │ (ingest pipeline warnings test-events.log)    │ PASS   │ 325.332333ms │
│ opencanary │ events      │ pipeline  │ (ingest pipeline warnings test-webhooks.json) │ PASS   │ 284.813959ms │
│ opencanary │ events      │ pipeline  │ test-events.log                               │ PASS   │  1.16716425s │
│ opencanary │ events      │ pipeline  │ test-webhooks.json                            │ PASS   │  45.164667ms │
╰────────────┴─────────────┴───────────┴───────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: opencanary - END   ---
Done
Run policy tests for the package
--- Test results for package: opencanary - START ---
No test results
--- Test results for package: opencanary - END   ---
Done
Run static tests for the package
--- Test results for package: opencanary - START ---
╭────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ opencanary │ events      │ static    │ Verify sample_event.json │ PASS   │  66.262833ms │
╰────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: opencanary - END   ---
Done
Run system tests for the package
2025/05/27 22:04:56  INFO License text found in "/SRC/GitHub/routedlogic/integrations/LICENSE.txt" will be included in package
2025/05/27 22:06:05  INFO Write container logs to file: /SRC/GitHub/routedlogic/integrations/build/container-logs/test-http_endpoint-1748347565803275000.log
2025/05/27 22:06:09  INFO Write container logs to file: /SRC/GitHub/routedlogic/integrations/build/container-logs/elastic-agent-1748347569388539000.log
2025/05/27 22:06:56  INFO Write container logs to file: /SRC/GitHub/routedlogic/integrations/build/container-logs/test-filestream-1748347616644031000.log
2025/05/27 22:06:59  INFO Write container logs to file: /SRC/GitHub/routedlogic/integrations/build/container-logs/elastic-agent-1748347619465036000.log
--- Test results for package: opencanary - START ---
╭────────────┬─────────────┬───────────┬───────────────┬────────┬───────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME     │ RESULT │  TIME ELAPSED │
├────────────┼─────────────┼───────────┼───────────────┼────────┼───────────────┤
│ opencanary │ events      │ system    │ filestream    │ PASS   │ 39.461239209s │
│ opencanary │ events      │ system    │ http_endpoint │ PASS   │ 56.612648666s │
╰────────────┴─────────────┴───────────┴───────────────┴────────┴───────────────╯
--- Test results for package: opencanary - END   ---
Done
user@box opencanary % docker ps | grep elastic
67dc3cc87104   docker.elastic.co/elastic-agent/elastic-agent-wolfi:8.17.0   "/usr/bin/tini -- /u…"   4 minutes ago   Up 3 minutes (healthy)   127.0.0.1:1514->1514/udp, 127.0.0.1:8082->80/tcp                 elastic-package-stack-elastic-agent-1
076a599d54ce   docker.elastic.co/elastic-agent/elastic-agent-wolfi:8.17.0   "/usr/bin/tini -- /u…"   4 minutes ago   Up 4 minutes (healthy)   127.0.0.1:8220->8220/tcp                                         elastic-package-stack-fleet-server-1
d45bb16814cf   docker.elastic.co/kibana/kibana:8.17.0                       "/bin/tini -- /usr/l…"   4 minutes ago   Up 4 minutes (healthy)   127.0.0.1:5601->5601/tcp                                         elastic-package-stack-kibana-1
ae8fbe84011f   docker.elastic.co/elasticsearch/elasticsearch:8.17.0         "/bin/tini -- /usr/l…"   4 minutes ago   Up 4 minutes (healthy)   127.0.0.1:9200->9200/tcp, 9300/tcp                               elastic-package-stack-elasticsearch-1
a08274c944ee   elastic-package-stack-package-registry                       "./package-registry"     4 minutes ago   Up 4 minutes (healthy)   127.0.0.1:8080->8080/tcp, 127.0.0.1:9000->9000/tcp               elastic-package-stack-package-registry-1
user@box opencanary %

efd6 · 2025-05-27T21:21:09Z

out of curiosity what's the situation that has you running tests on an 8.13.x stack? Did you explicitly look at the constraint and fire up an 8.13.x version to test with because it had 8.13.0?

CI runs the tests on the lowest claimed stack version specified in the kibana.version field in the manifest. So, yes, that's what I did.

Since it works for 8.15, I'd be happy for it to be marked at that version. This drops the least number of users.

navnit-elastic · 2025-06-02T05:04:01Z

/test

navnit-elastic · 2025-06-02T05:16:34Z

Hello @colin-stubbs, Recently, there have been changes to the opencanary integration (here). can you please sync this branch with main to resolve the conflicts?

Add VNC auth events including password fields, improve password redaction using gsub

colin-stubbs · 2025-06-03T12:13:58Z

Hello @colin-stubbs, Recently, there have been changes to the opencanary integration (here). can you please sync this branch with main to resolve the conflicts?

Sorted.

navnit-elastic · 2025-06-03T12:21:41Z

/test

all tests pass

colin-stubbs · 2025-06-03T12:22:45Z

@navnit-elastic apologies, I just did another commit to reduce the new kibana constraint to 8.15.0 as efd6 suggested.

fully re-tested using elastic-package on an 8.15.0 stack.

navnit-elastic · 2025-06-03T12:24:24Z

No problem and thank you for the changes!

elastic-vault-github-plugin-prod · 2025-06-03T12:47:18Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6 · 2025-06-03T23:19:01Z

/test

packages/opencanary/changelog.yml

packages/opencanary/manifest.yml

packages/opencanary/data_stream/events/_dev/test/pipeline/test-events.log

packages/opencanary/data_stream/events/elasticsearch/ingest_pipeline/default.yml

kcreddy · 2025-06-11T06:54:52Z

/test

packages/opencanary/manifest.yml

efd6

Please also address #13970 (comment) and #13970 (comment).

packages/opencanary/changelog.yml

colin-stubbs · 2025-06-17T07:41:56Z

@efd6 I've reverted as above.

re: your second comment reference, that one's still in your court from my perspective given you have the issue with respect to having test events that apparently need to be reviewed line by line by a human.

From my perspective a selection of event types should be as complete as possible, as it is valuable in both system testing and pipeline testing to detect undefined fields, field type conflicts, and any beat level ingest or pipeline processing errors.

The set of events that is in there is one that I have already trimmed of duplicates that were generated during testing. There are many similar events, but they all have different content.

At the moment, and in response to your comment, I've already cut the test events for the pipeline tests down to one single event for filestream, and one single event for http_endpoint/webhooks.

The system tests still have the larger selection of 161 events, and the system tests validate that 161 events wind up in the appropriate index via assert.hit_count.

It's my understanding that both pipeline test and system test results are checked for common errors/problems by elastic-package.

What do you want to do?

navnit-elastic · 2025-06-17T10:17:55Z

@colin-stubbs, I share Dan's point here. A few events in the system tests could serve the purpose of checking for any data collection errors through the filestream input.
Cases such as missing field definition and type conflicts can be detected through the pipeline tests. hence, it is better to include variants of events in the pipeline tests rather than the system tests.
Also, the pipeline tests can potentially be reduced here. For example, by looking at the diff between two events in the pipeline tests:

--- 1.json      2025-06-17 15:27:32.413184250 +0530
+++ 2.json      2025-06-17 15:27:41.579156442 +0530
@@ -1,16 +1,16 @@
 {
     "dst_host": "",
     "dst_port": -1,
-    "local_time": "2024-04-03 20:58:39.605621",
-    "local_time_adjusted": "2024-04-03 14:58:39.605642",
+    "local_time": "2024-04-03 20:58:39.605904",
+    "local_time_adjusted": "2024-04-03 14:58:39.605919",
     "logdata": {
         "msg": {
-            "logdata": "Added service from class CanaryFTP in opencanary.modules.ftp to fake"
+            "logdata": "Added service from class CanaryGit in opencanary.modules.git to fake"
         }
     },
     "logtype": 1001,
     "node_id": "opencanary-1",
     "src_host": "",
     "src_port": -1,
-    "utc_time": "2024-04-03 20:58:39.605638"
+    "utc_time": "2024-04-03 20:58:39.605916"
 }

These events only contain changes in field values and does not necessarily pass through the pipeline differently.

colin-stubbs · 2025-06-17T10:53:52Z

@colin-stubbs, I share Dan's point here. A few events in the system tests could serve the purpose of checking for any data collection errors through the filestream input. Cases such as missing field definition and type conflicts can be detected through the pipeline tests. hence, it is better to include variants of events in the pipeline tests rather than the system tests. Also, the pipeline tests can potentially be reduced here. For example, by looking at the diff between two events in the pipeline tests:

These events only contain changes in field values and does not necessarily pass through the pipeline differently.

OK, fair points, it looks like it could actually go down to 34 events.

navnit-elastic · 2025-06-17T13:15:28Z

@colin-stubbs, Thanks for the commit, but we would like the pipeline tests to cover those events and only a few in the system tests.

colin-stubbs · 2025-06-18T05:17:07Z

@colin-stubbs, Thanks for the commit, but we would like the pipeline tests to cover those events and only a few in the system tests.

Righteyohthen.

colin-stubbs · 2025-07-01T04:23:51Z

@navnit-elastic @efd6 the world turns and the PR is becoming problematic, I've merged from main again as someone else has modified opencanary with version 0.6.0 so now bumped this one to 0.6.1.

What is the problem with releasing this?

efd6 · 2025-07-01T04:36:38Z

packages/opencanary/data_stream/events/_dev/test/pipeline/test-events.log

+{"dst_host": "1.128.0.1", "dst_port": 8001, "local_time": "2025-05-21 02:39:50.857806", "local_time_adjusted": "2025-05-21 02:39:50.857820", "logdata": {"BANNER_ID": "1", "DATA": "HELP", "FUNCTION": "DATA_RECEIVED"}, "logtype": 18004, "node_id": "opencanary-1", "src_host": "1.128.0.10", "src_port": 41072, "utc_time": "2025-05-21 02:39:50.857816"}
+{"dst_host": "1.128.0.1", "dst_port": 8001, "local_time": "2025-05-21 02:42:42.866533", "local_time_adjusted": "2025-05-21 02:42:42.866540", "logdata": {"BANNER_ID": "1", "DATA": "\u0080\u0000\u0000(\u001d\u00f2\u00cb\\\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0002\u0000\u0001\u0086\u00a0\u0000\u0000\u0000\u0002\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000", "FUNCTION": "DATA_RECEIVED"}, "logtype": 18004, "node_id": "opencanary-1", "src_host": "1.128.0.10", "src_port": 348, "utc_time": "2025-05-21 02:42:42.866538"}
+{"dst_host": "1.128.0.1", "dst_port": 5000, "local_time": "2025-06-03 10:56:41.899284", "local_time_adjusted": "2025-06-03 20:56:41.899304", "logdata": {"VNC Client Response": "ef39a47fc14af31260d11bc13da7612c", "VNC Password": "<Password was not in the common list>", "VNC Server Challenge": "1ef1f96bd2d7cc5bd1e85593d17401b7"}, "logtype": 12001, "node_id": "opencanary-1", "src_host": "1.128.0.10", "src_port": 52668, "utc_time": "2025-06-03 10:56:41.899300"}
+{"dst_host": "1.128.0.1", "dst_port": 5000, "local_time": "2025-06-03 10:58:03.263583", "local_time_adjusted": "2025-06-03 20:58:03.263633", "logdata": {"VNC Client Response": "406f0aee0ca9cf1fd61916950610fb59", "VNC Password": "password", "VNC Server Challenge": "f99f85ee50d99d748c10e3217f2a4e57"}, "logtype": 12001, "node_id": "opencanary-1", "src_host": "1.128.0.10", "src_port": 53326, "utc_time": "2025-06-03 10:58:03.263628"}


You know what I'm going to say here. I'd fix it, but maintainers have not been given edit right on this PR. I will ignore it for the sake of sanity.

efd6 · 2025-07-01T04:47:40Z

/test

elasticmachine · 2025-07-01T05:12:52Z

💚 Build Succeeded

Buildkite Build
Commit: c2efe54

History

💚 Build #27100 succeeded 9609437
💚 Build #26834 succeeded 399acd5
💚 Build #26802 succeeded b8e4cfb
💔 Build #26697 failed 3a89b33
💔 Build #26398 failed 1d0c6a2

elastic-sonarqube · 2025-07-01T05:12:59Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

efd6

Thanks

elastic-vault-github-plugin-prod · 2025-07-01T05:57:09Z

Package opencanary - 0.6.1 containing this change is available at https://epr.elastic.co/package/opencanary/0.6.1/

* Utilise filestream take_over option to ease migration from log to filestream inputs * Improve `event.kind` field mapping. * Add OpenCanary webhook support. * Migrate logfile input to filestream. * Make password redaction configurable. * Make retention of custom fields mapped to ECS fields configurable. * Improve OpenCanary field mapping. * Add dashboard. * Fix confused tftp/vnc field names.

colin-stubbs added 2 commits May 20, 2025 22:29

rebase fork from elastic/integrations:main to rebuild and retest

0bb23a1

test using latest opencanary version

1d0c6a2

rebuild test event set, fix handling for certain error type events, add GeoIP enrichment for source.ip and destination.ip, rebuild/retest/regen sample events etc, retest dashboard

colin-stubbs requested a review from a team as a code owner May 22, 2025 13:45

andrewkroh added Integration:opencanary OpenCanary (Community supported) dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels May 22, 2025

update kibana constraint

3a89b33

colin-stubbs added 2 commits June 3, 2025 22:09

merge main to opencanary fork, resolve conflicts, full retest

976933f

Add VNC auth events including password fields, improve password redaction using gsub

Merge remote-tracking branch 'origin/main' into opencanary

b8e4cfb

full retest on 8.15.0, drop kibana version constraint to 8.15.0

399acd5

all tests pass

efd6 reviewed Jun 4, 2025

View reviewed changes

apply review suggestions, bump version to 0.6.0

9609437

kcreddy reviewed Jun 11, 2025

View reviewed changes

packages/opencanary/manifest.yml Show resolved Hide resolved

colin-stubbs added 2 commits June 17, 2025 11:33

Merge branch 'main' into opencanary

93286b8

utilise take_over functionality for logstream, bump version

ac234ae

efd6 reviewed Jun 17, 2025

View reviewed changes

packages/opencanary/changelog.yml Show resolved Hide resolved

revert to 0.6.0, expand changelog entries as recommended

fe5e62d

reduce test event count

1f39212

swap the majority of test events back to pipeline tests

8365cf8

colin-stubbs added 2 commits July 1, 2025 14:21

bump to 0.6.1 in order to merge main from source

4e18ac1

Merge remote-tracking branch 'origin/main' into opencanary

c2efe54

efd6 reviewed Jul 1, 2025

View reviewed changes

efd6 approved these changes Jul 1, 2025

View reviewed changes

efd6 merged commit 802b11e into elastic:main Jul 1, 2025
7 checks passed

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025

Opencanary #13970

Opencanary #13970

Uh oh!

Conversation

colin-stubbs commented May 22, 2025

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

colin-stubbs commented May 22, 2025

Uh oh!

elasticmachine commented May 22, 2025

Uh oh!

navnit-elastic commented May 26, 2025

Uh oh!

kcreddy commented May 26, 2025

Uh oh!

efd6 commented May 26, 2025

Uh oh!

navnit-elastic commented May 27, 2025

Uh oh!

colin-stubbs commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efd6 commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

navnit-elastic commented Jun 2, 2025

Uh oh!

navnit-elastic commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

colin-stubbs commented Jun 3, 2025

Uh oh!

navnit-elastic commented Jun 3, 2025

Uh oh!

colin-stubbs commented Jun 3, 2025

Uh oh!

navnit-elastic commented Jun 3, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jun 3, 2025

🚀 Benchmarks report

Uh oh!

efd6 commented Jun 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy commented Jun 11, 2025

Uh oh!

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

colin-stubbs commented Jun 17, 2025

Uh oh!

navnit-elastic commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

colin-stubbs commented Jun 17, 2025

Uh oh!

navnit-elastic commented Jun 17, 2025

Uh oh!

colin-stubbs commented Jun 18, 2025

Uh oh!

colin-stubbs commented Jul 1, 2025

Uh oh!

efd6 Jul 1, 2025

Choose a reason for hiding this comment

colin-stubbs commented May 27, 2025 •

edited

Loading

efd6 commented May 27, 2025 •

edited

Loading

navnit-elastic commented Jun 2, 2025 •

edited

Loading

navnit-elastic commented Jun 17, 2025 •

edited

Loading