[Logstash] Add CEL native data collection, and dashboards for Pipeline and Pipeline Details #8098

robbavey · 2023-10-04T20:59:29Z

Proposed commit message

This PR follows on from #7704, and adds data collection to the integration, as well as standard Dashboards for Logstash Pipelines.

The short term goal for this is to enable Logstash to be monitored with Serverless Elasticsearch instances, which do not provide a Stack Monitoring UI, with the long term goal of iterating on this to replace the existing Stack Monitoring UI.

This PR includes 3 new data streams - node_cel, pipelines and plugins, with data collected from the Logstash Metrics API using the CEL input, intended to replace the existing metricbeat collection.

This give us more flexibility going forward, and should improve our velocity to add new features to the Logstash dashboards straight from the API, without needing to update several code bases.

The CEL data streams will create a single document per node for the node_cel data stream, a document per pipeline per node for the pipeline data stream, and a document per plugin per pipeline per node for the plugins data stream. For this reason, we default to a lower sending cadence for the plugins data stream than for the other.

Note that the existing data collection from metricbeat defaults to sending large documents, including significant data for the entire pipeline graph every 10 seconds.

This integration also includes ingest pipelines to clean up the data from the Logstash flow metrics API, to remove entries we don't necessarily need at this stage, and to remove the string representation of "Infinity".

When installing a Logstash integration, users will be presented with an option to collect Metrics for Stack Monitoring or for Technical Preview

(Note: If there was a way to have an either/or option in the UI, that would be very helpful)

The dashboards are described in the Screenshots section.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Install elastic-package
Checkout this PR
Navigate to the integrations/packages/logstash directory
Run elastic-package build to build the package
Run elastic-package stack up to bring up the Elastic stack
Start an instance of Logstash
Install the Logstash integration
Uncheck the Metrics (Stack Monitoring) option
Check the Metrics (Technical Preview) option
Click 'Save and Continue' and follow the instructions to enroll the agent

Related issues

Follows on from #7704

Screenshots

Integration Selection Page

Pipelines Dashboard

This Dashboard provides an overview of running pipelines, with a table containing details of pipelines running, which can be drilled down into a "details view"

Proposed Page

This is intended to replace the existing pipelines page in the Stack monitoring solution

Existing Page

The page includes additional details, including Persistent Queue usage per pipeline, and more visibility into which pipelines may be blocked

Pipeline Details Dashboard

The Pipeline Details dashboard takes a different approach to the pipeline details view in stack monitoring

The page includes some general pipeline information, showing the rate of events flowing through the pipelines and state of persistent queues, broken down per node running the pipeline:

Proposed Page

General Information

It also includes information for each plugin type:

Inputs

Filters

Outputs

Codecs

This has benefits and drawbacks compared with the existing approach:

The benefits are:

More information and more rich drill downs to help track down slow performing plugins
Can track PQ usage on a per-pipeline and per-node basis

Drawbacks:

The lack of visual representation may make it more difficult to figure out which plugin is at fault when multiple plugins of the same type are at play
- Note - this can be mitigated by users including an id in their plugin definition
- We will be looking to add more information (originating file + line number of plugin) in a subsequent follow-up PR.

Existing Dashboard

The existing dashboard contains a representation of the pipeline graph, which we are not easily able to show using standard Kibana Dashboards

And a small pop-up drilldown for each plugin

Note: This will require a follow-up PR to enable for serverless - the existing integration has some compatibility issues that I would like to fold into a subsequent PR

This commit adds infrastructure to retrieve pipeline information from the Logstash Metrics API using CEL. This is intended to provide a flattened version of the data provided by the metrics API enabling visualizations to be created using standard Kibana dashboards, rather than rely on the custom stack monitoring UI

This commit adds infrastructure to retrieve plugin information from the Logstash Metrics API using CEL. This is intended to provide a flattened version of the data provided by the metrics API enabling visualizations to be created using standard Kibana dashboards, rather than rely on the custom stack monitoring UI. Note that by default, we are sending data at a slower cadence than for nodes and pipelines. Large Logstash pipelines may include *many* plugins, to avoid excessive data transmission, we will send at a lower cadence, and allow users to adjust depending on their needs

This commit adds infrastructure to retrieve node information from the Logstash Metrics API using CEL. This is intended to replace the existing node retrieval for stack monitoring and is intended to mirror the data that was previously retrieved by metricbeat

This commit adds dashboards for Logstash pipelines and plugins

Fix tests

elasticmachine · 2023-10-04T21:14:32Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-10-09T18:01:25.647+0000
Duration: 22 min 1 sec

Test stats 🧪

Test	Results
Failed	0
Passed	40
Skipped	0
Total	40

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

robbavey · 2023-10-04T21:56:12Z

/test

elasticmachine · 2023-10-05T00:37:10Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`6/6`)	💚
Classes	100.0% (`6/6`)	💚
Methods	100.0% (`55/55`)	💚 36.585
Lines	87.558% (`190/217`)	👎 -12.306
Conditionals	100.0% (`0/0`)	💚

andrewkroh · 2023-10-05T02:23:55Z

packages/logstash/data_stream/node_cel/agent/stream/stream.yml.hbs

My personal preference is to name this file based on the input type used like cel.yml.hbs. If you do so, then update the template_path attribute in the manifest.yml.

andrewkroh · 2023-10-05T02:26:06Z

packages/logstash/data_stream/node_cel/agent/stream/stream.yml.hbs

+  password: {{password}}
+
+redact:
+  fields: ~


Suggested change

fields: ~

fields:

- password

But if you switch to using auth.basic then you won't need to redact it or configure the request with the Authorization header manually.

auth: basic: user: {{escape_string username}} password: {{escape_string password}}

andrewkroh · 2023-10-05T02:33:43Z

packages/logstash/validation.yml

@@ -0,0 +1,3 @@
+errors:
+  exclude_checks:
+    - SVR00002  # Mandatory filters in dashboards.


I think this needs an elastic-package format.

I'm going to remove for this now - I will add a follow up PR for serverless/3.0 format compatibility

andrewkroh · 2023-10-05T02:44:38Z

packages/logstash/data_stream/node_cel/fields/fields.yml

+                    - name: cpuacct
+                      type: group
+                      fields:
+                        - name: control_group


There's a little inconsistency between node_cel and node_stats. control_group is indexed as text in at packages/logstash/data_stream/node_stats/fields/fields.yml:128:27. Same for the *.cpu.control_group.

That might cause some query/agg issues.

andrewkroh · 2023-10-05T02:47:37Z

packages/logstash/data_stream/node_cel/fields/agent.yml

+      type: keyword
+      ignore_above: 1024
+      description: Host mac addresses.
+    - name: name


This field is already declared in ecs.yml.

andrewkroh · 2023-10-05T02:52:45Z

packages/logstash/data_stream/node_cel/fields/agent.yml

+    - name: account.id
+      type: keyword
+      ignore_above: 1024
+      description: "The cloud account or organization id used to identify different entities in a multi-tenant environment.\nExamples: AWS account id, Google Cloud ORG Id, or other unique identifier."


A lot of these fields exist in ECS and should use external: ecs.

Suggested change

- name: account.id

type: keyword

ignore_above: 1024

description: "The cloud account or organization id used to identify different entities in a multi-tenant environment.\nExamples: AWS account id, Google Cloud ORG Id, or other unique identifier."

- name: account.id

external: ecs

I've been working on a tool to fix this automatically. This command will edit the file for you. (I would only use it on a clean workspace so you can review its edits in isolation.)

go run github.com/andrewkroh/fydler@main -fix packages/logstash/data_stream/node_cel/fields/*yml

efd6 · 2023-10-05T03:06:57Z

packages/logstash/data_stream/node_cel/agent/stream/stream.yml.hbs

+    get_request(state.url).with({
+        "Header":{
+            "Authorization": ["Basic "+string(base64(state.username+":"+state.password))],
+        }
+    })


This can be done more easily withe the basic_authentication extension.

Is there something I am missing beyond adding auth.basic.user and auth.basic.password blocks eg -

config_version: "2" interval: {{period}} resource.url: "{{url}}/_node/stats?graph=true&vertices=true" {{#if resource_ssl}} resource.ssl: {{resource_ssl}} {{/if}} {{#if username}} auth.basic.user: {{escape_string username}} {{/if}} {{#if password}} auth.basic.password: {{escape_string password}} {{/if}} redact: fields: ~ program: | get_request(state.url) .do_request().as(resp, bytes(resp.Body) : :

I have tried this a number of times, and I've never seen the Authorization header sent successfully, hence the workarounds applied here.

efd6 · 2023-10-05T03:14:44Z

packages/logstash/data_stream/node_cel/agent/stream/stream.yml.hbs

+             "events":body['events'],
+             "jvm":{
+                "uptime_in_millis":body['jvm']['uptime_in_millis'],


These and below can be dotted, eg:

Suggested change

"events":body['events'],

"jvm":{

"uptime_in_millis":body['jvm']['uptime_in_millis'],

"events":body.events,

"jvm":{

"uptime_in_millis":body.jvm.uptime_in_millis,

Stylistic only, but slightly lighter on the page.

efd6 · 2023-10-05T03:20:12Z

packages/logstash/data_stream/node_cel/agent/stream/stream.yml.hbs

+        }})
+  )
+  .as(eve, {
+    "events":eve.map(each, eve),


What is the intention here? It looks to me like you are wanting a single element array with eve as the element. Is that correct?

Suggested change

"events":eve.map(each, eve),

"events": [eve],

In this case, yes - your suggestion works perfectly

efd6 · 2023-10-05T03:24:16Z

packages/logstash/data_stream/plugins/agent/stream/stream.yml.hbs

+      .with({
+        "es_cluster_id":((body.pipelines[pipeline_name].vertices).as(vertices, vertices.map(each, has(each.cluster_uuid), each.cluster_uuid)))
+      })
+      .with({
+        "es_cluster_id_map":((body.pipelines[pipeline_name].vertices).as(vertices, vertices.map(each, has(each.cluster_uuid), {"plugin_id":each.id, "cluster_id":each.cluster_uuid})))
+      })
+      .with({
+        "outputs":body.pipelines[pipeline_name]["plugins"]["outputs"]
+      })
+      .with({
+        "inputs":body.pipelines[pipeline_name]["plugins"]["inputs"]
+      })
+      .with({
+        "filters":body.pipelines[pipeline_name]["plugins"]["filters"]
+      })
+      .with({
+        "codecs":body.pipelines[pipeline_name]["plugins"]["codecs"]
+      })
+      .with({
+        "host":{
+          "name":body.name,
+          "address":body.http_address,
+        }
+      })


Each with call will need to allocate a Go map. I would combine all these into one.

Suggested change

.with({

"es_cluster_id":((body.pipelines[pipeline_name].vertices).as(vertices, vertices.map(each, has(each.cluster_uuid), each.cluster_uuid)))

})

.with({

"es_cluster_id_map":((body.pipelines[pipeline_name].vertices).as(vertices, vertices.map(each, has(each.cluster_uuid), {"plugin_id":each.id, "cluster_id":each.cluster_uuid})))

})

.with({

"outputs":body.pipelines[pipeline_name]["plugins"]["outputs"]

})

.with({

"inputs":body.pipelines[pipeline_name]["plugins"]["inputs"]

})

.with({

"filters":body.pipelines[pipeline_name]["plugins"]["filters"]

})

.with({

"codecs":body.pipelines[pipeline_name]["plugins"]["codecs"]

})

.with({

"host":{

"name":body.name,

"address":body.http_address,

}

})

.with({

"es_cluster_id":((body.pipelines[pipeline_name].vertices).as(vertices, vertices.map(each, has(each.cluster_uuid), each.cluster_uuid))),

"es_cluster_id_map":((body.pipelines[pipeline_name].vertices).as(vertices, vertices.map(each, has(each.cluster_uuid), {"plugin_id":each.id, "cluster_id":each.cluster_uuid}))),

"outputs":body.pipelines[pipeline_name].plugins.outputs,

"inputs":body.pipelines[pipeline_name].plugins.inputs,

"filters":body.pipelines[pipeline_name].plugins.filters,

"codecs":body.pipelines[pipeline_name].plugins.codecs,

"host":{

"name":body.name,

"address":body.http_address,

}

})

Maybe also consider pulling out body.pipelines[pipeline_name].plugins once, before the evaluation here.

(oops, missed the initial with, but you get the idea)

efd6 · 2023-10-05T03:26:16Z

packages/logstash/data_stream/plugins/agent/stream/stream.yml.hbs

+                "ms":input.events.queue_push_duration_in_millis
+              }
+            }
+          }.drop_empty()


Check to see that you still get what you want with only the final drop_empty (L87) in the expression.

efd6 · 2023-10-05T03:27:10Z

packages/logstash/data_stream/plugins/agent/stream/stream.yml.hbs

+                "ms":filter.events.duration_in_millis
+              }
+            }
+          }.drop_empty()


Similar here.

packages/logstash/data_stream/plugins/agent/stream/stream.yml.hbs

robbavey · 2023-10-05T20:11:15Z

I think I've addressed most of the comments in the original reviews - apart from the simplified basic_auth, which I have been unable to get working thus far

Using the `get()` method instead of `get_request(url).do_request()` enables the use of basic auth by simply specifying `auth.basic` in the yaml definition allowing a whole bunch of code to be removed from the script, and only needing to redact the passwords, rather than use a blanket redaction.

efd6 · 2023-10-05T21:41:32Z

packages/logstash/data_stream/pipeline/agent/stream/cel.yml.hbs

+      .with({
+        "host":{
+          "name":body.name,
+          "address":body.http_address,
+        }
+      })
+      .with({


You can merge these as well.

efd6 · 2023-10-05T21:42:45Z

packages/logstash/data_stream/pipeline/agent/stream/cel.yml.hbs

+          "flow":body.pipelines[pipeline_name].flow,
+          "time":{
+            "queue_push_duration":{
+              "ms":body.pipelines[pipeline_name]["events"]["queue_push_duration_in_millis"],


Suggested change

"ms":body.pipelines[pipeline_name]["events"]["queue_push_duration_in_millis"],

"ms":body.pipelines[pipeline_name].events.queue_push_duration_in_millis,

(similar below — quoted bracket indexing is only needed when the key is not a valid CEL token)

efd6 · 2023-10-05T21:44:08Z

packages/logstash/data_stream/pipeline/agent/stream/cel.yml.hbs

+  fields:
+    - password


This can now be the null case.

Suggested change

fields:

- password

fields: ~

efd6 · 2023-10-05T21:44:43Z

packages/logstash/data_stream/plugins/agent/stream/cel.yml.hbs

+  fields:
+    - password


Suggested change

fields:

- password

fields: ~

efd6 · 2023-10-05T21:45:23Z

packages/logstash/data_stream/plugins/agent/stream/cel.yml.hbs

+      .with({
+        "es_cluster_id":((body.pipelines[pipeline_name].vertices).as(vertices, vertices.map(each, has(each.cluster_uuid), each.cluster_uuid))),
+        "es_cluster_id_map":((body.pipelines[pipeline_name].vertices).as(vertices, vertices.map(each, has(each.cluster_uuid), {"plugin_id":each.id, "cluster_id":each.cluster_uuid}))),
+        "outputs":body.pipelines[pipeline_name]["plugins"]["outputs"],


Suggested change

"outputs":body.pipelines[pipeline_name]["plugins"]["outputs"],

"outputs":body.pipelines[pipeline_name].plugins.outputs,

(similar below)

efd6 · 2023-10-05T21:45:52Z

packages/logstash/data_stream/plugins/agent/stream/cel.yml.hbs

+        "plugin":{
+          "type":"input",
+          "input":{
+            "elasticsearch.cluster.id":event['es_cluster_id_map'].map(tuple, (tuple.plugin_id == input.id), tuple.cluster_id),


Suggested change

"elasticsearch.cluster.id":event['es_cluster_id_map'].map(tuple, (tuple.plugin_id == input.id), tuple.cluster_id),

"elasticsearch.cluster.id":event.es_cluster_id_map.map(tuple, (tuple.plugin_id == input.id), tuple.cluster_id),

(similar below)

robbavey · 2023-10-06T00:54:08Z

This is ready for another round

efd6

Minor nits, then LGTM for the CEL code.

packages/logstash/data_stream/pipeline/agent/stream/cel.yml.hbs

efd6 · 2023-10-06T01:37:34Z

packages/logstash/data_stream/plugins/agent/stream/cel.yml.hbs

+          "filter":{
+            "id":filter.id,
+            "name":filter.name,
+            "elasticsearch.cluster.id":event['es_cluster_id_map'].map(tuple, (tuple.plugin_id == filter.id), tuple.cluster_id),


Suggested change

"elasticsearch.cluster.id":event['es_cluster_id_map'].map(tuple, (tuple.plugin_id == filter.id), tuple.cluster_id),

"elasticsearch.cluster.id":event.es_cluster_id_map..map(tuple, (tuple.plugin_id == filter.id), tuple.cluster_id),

…e in filter view

robbavey · 2023-10-09T13:38:46Z

This is ready for another round @elastic/infra-monitoring-ui

Note, that this is part of the serverless initiative to enable Logstash monitoring in the absence of a stack monitoring UI

I will be following on this PR with the changes necessary to make this package visible to serverless, which includes changes to the existing integration package

Result of running: go run github.com/andrewkroh/fydler@main -fix packages/logstash/data_stream/*/fields/*yml

klacabane

LGTM!

IanLee1521 · 2023-11-30T06:54:09Z

First off, I just found this and it's AWESOME!

Maybe it's better for me to open a new issue, but just before I do, would it make sense to mark these dashboards with a reference to users needing to switch the integration from Stack Monitoring to Technical Preview?

I just ran in to this where I was trying to look at the [Metrics Logstash] * dashboards and getting errors.

Eventually I found this need to change the integration configuration and now things are working.

Thoughts? Is that worth noting on the dashboards?

robbavey · 2023-12-01T15:48:31Z

@IanLee1521 First of all, thanks for trying out the new dashboards in technical preview - we love to get feedback on new features we are developing, and we appreciate you trying them out as we refine them. Any further comments would be gratefully received.

I think this should be an issue - as we start to transition to the new way of collecting and presenting data for monitoring, we should be making it clearer how to work with this effectively.

I'm on PTO at the moment, but I can make an issue when I get back, unless you would prefer to create one yourself.

robbavey · 2023-12-04T16:04:19Z

@IanLee1521 - I've added an issue here. And thanks again for the feedback!

robbavey added 7 commits October 3, 2023 09:32

Add CEL node data collection

a13ff6b

This commit adds infrastructure to retrieve node information from the Logstash Metrics API using CEL. This is intended to replace the existing node retrieval for stack monitoring and is intended to mirror the data that was previously retrieved by metricbeat

Add dashboards for pipelines and plugins

f2584d1

This commit adds dashboards for Logstash pipelines and plugins

Update existing dashboards to include pipeline and plugin references

55b7bb1

Tweaks and fixes

780f665

Fix tests

Tidy and use time series indices

e5d96f0

robbavey added the enhancement New feature or request label Oct 4, 2023

robbavey requested a review from a team as a code owner October 4, 2023 20:59

robbavey force-pushed the logstash_cel branch from 934b086 to 4402f84 Compare October 4, 2023 22:20

Tidying, adding readme text

f801aa7

robbavey force-pushed the logstash_cel branch from 4402f84 to f801aa7 Compare October 4, 2023 22:36

Bump required kibana to 8.10.1

48bd0e8

andrewkroh reviewed Oct 5, 2023

View reviewed changes

efd6 reviewed Oct 5, 2023

View reviewed changes

robbavey added 8 commits October 5, 2023 09:37

Rename stream.yml.hbs to cel.yml.hbs

f2c8d43

Simlify mapping into events field in CEL scripts

0d6b939

Tidy code by using dot notation

596f0d8

Tidy up Dashboard

7909f7e

Clean up 'with' functions

8cb394b

Set fields that already exist in ECS to external: ECS

3e71129

Make control group field types consistent

ad027b3

Remove unnecessary drop_empty() commands

a4ba6b4

efd6 reviewed Oct 5, 2023

View reviewed changes

Futher tidying after code review comments

4fa4eac

efd6 approved these changes Oct 6, 2023

View reviewed changes

robbavey added 5 commits October 6, 2023 09:41

Style changes to cel scripts

ffd380c

Remove logstash monitoring pipeline from node view

7c2e47e

Polish dashboards

d47de67

Additional polish to dashboards

637eda9

Fix typo in nav panel, and fix flow metric to show correct plugin typ…

2e934fc

…e in filter view

robbavey force-pushed the logstash_cel branch from 7b04009 to f2c776e Compare October 9, 2023 15:41

remove duplicate fields

559a8ee

Result of running: go run github.com/andrewkroh/fydler@main -fix packages/logstash/data_stream/*/fields/*yml

robbavey force-pushed the logstash_cel branch from f2c776e to 559a8ee Compare October 9, 2023 18:01

klacabane approved these changes Oct 10, 2023

View reviewed changes

robbavey merged commit e089669 into elastic:main Oct 10, 2023

robbavey mentioned this pull request Oct 11, 2023

[Logstash] Update manifest format_version to 3.0.0 for logstash integ… #8166

Merged

4 tasks

robbavey mentioned this pull request Dec 4, 2023

[Logstash] Add note to Nav Panel explaining need to enable CEL data collection for metrics dashboards to work #8643

Open

andrewkroh added the Integration:logstash Logstash label Jul 22, 2024

	"ms":body.pipelines[pipeline_name]["events"]["queue_push_duration_in_millis"],
	"ms":body.pipelines[pipeline_name].events.queue_push_duration_in_millis,

	"outputs":body.pipelines[pipeline_name]["plugins"]["outputs"],
	"outputs":body.pipelines[pipeline_name].plugins.outputs,

	"elasticsearch.cluster.id":event['es_cluster_id_map'].map(tuple, (tuple.plugin_id == input.id), tuple.cluster_id),
	"elasticsearch.cluster.id":event.es_cluster_id_map.map(tuple, (tuple.plugin_id == input.id), tuple.cluster_id),

[Logstash] Add CEL native data collection, and dashboards for Pipeline and Pipeline Details #8098

[Logstash] Add CEL native data collection, and dashboards for Pipeline and Pipeline Details #8098

Uh oh!

Conversation

robbavey commented Oct 4, 2023

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Pipelines Dashboard

Proposed Page

Existing Page

Pipeline Details Dashboard

Proposed Page

General Information

Inputs

Filters

Outputs

Codecs

Existing Dashboard

Uh oh!

elasticmachine commented Oct 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

robbavey commented Oct 4, 2023

Uh oh!

elasticmachine commented Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewkroh Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

robbavey commented Oct 5, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Oct 4, 2023 •

edited

Loading

elasticmachine commented Oct 5, 2023 •

edited

Loading

andrewkroh Oct 5, 2023 •

edited

Loading