system: convert visualisations to lens #5740

efd6 · 2023-03-30T01:44:49Z

What does this PR do?

Converts visualisations to Lens where possible.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Caveats:

No "Convert to Lens" option was available for "Logon Sources [Windows System Security]" in User Logon Information
Word cloud, "Target Users [Windows System Security]", not converted in User Management Events
Word cloud "Group Management Events - Target Groups - Tag Cloud [Windows System Security]" not converted in Group Management Events
In the "Group Management Events - Groups vs Actions - Heatmap [Windows System Security]" the horizontal axis is visually broken because labels may not be rotated or omitted

Concerns:

I am concerned about the application of this change since it requires all users to be on 8.7 to pick up any future bug fixes.

How to test this PR locally

Related issues

For [System] Dashboard clean-up for Windows event logs #5022

Screenshots

Overview

User Logon Information

Logon Failed and Account Lockout

fold

User Management Events

fold

Group Management Events

fold

elasticmachine · 2023-03-30T02:00:09Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-04-30T21:50:34.474+0000
Duration: 16 min 39 sec

Test stats 🧪

Test	Results
Failed	0
Passed	145
Skipped	0
Total	145

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-03-30T04:39:00Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`3/3`)	💚
Files	100.0% (`4/4`)	💚 3.035
Classes	100.0% (`4/4`)	💚 3.035
Methods	60.759% (`48/79`)	👎 -30.934
Lines	99.535% (`2780/2793`)	👍 7.492
Conditionals	100.0% (`0/0`)	💚

rdner · 2023-03-30T07:12:20Z

LGTM but I'd like @cmacknz to have a look as well, I don't have much experience with this.

efd6 · 2023-03-30T07:25:41Z

@rdner Agreed, I'd also like to wait for the kibana people too. @drewdaemon Can you take a look?

cmacknz · 2023-03-30T19:16:28Z

Not a visualization expert, don't wait for my approval on this.

drewdaemon

Generally speaking, this LGTM and I'm certainly happy to see this PR!

To state it "out-loud," this PR is limited to the following dashboards

[System Windows Security] User Management Events
[System] Windows Overview
[System Windows Security] User Logons
[System Windows Security] Group Management Events
[System Windows Security] Failed and Blocked Accounts

A few comments

I see that you are using the Legacy Metric Lens visualization. We recommend using the new Metric visualization instead, since the Legacy Metric will eventually be deprecated.
We generally recommend replacing tag clouds with horizontal bar charts to allow you to convert to Lens. See this comment for more info.
The description for the dashboard called [System Windows Security] Failed and Blocked Accounts includes mention of TSVB. Can we remove that?

I see that an old palette got transferred over automatically by the convert-to-Lens function in many of the visualizations. If there's no objection, could we change all those palettes to the current system default?

Old palette:

Default palette

efd6 · 2023-03-31T01:59:44Z

@drewdaemon Queries:

Is there a way to convert a tag cloud to a histogram? I don't see an option for this. I've done it manually, but this seems to have a pretty significant impact on the semantics of the chart.
These are adjacent visualisations looking at the same data with the same filters (none):
It seems that the Compatibility palette is called "kibana_palette" in the JSON. Is it valid to just replace that name with "default" in the JSON directly? Digging through the UI to replace 20 instance of this seems sort of inefficient.
The choice of staying with the legacy metric was because of the increased font size for this compared to the alternative. Is it possible to increase font size to use more/waste less of the available panel in the new metric visualisation?

drewdaemon · 2023-03-31T14:26:30Z

@efd6

Is there a way to convert a tag cloud to a histogram? I don't see an option for this. I've done it manually, but this seems to have a pretty significant impact on the semantics of the chart.

No, you have to do it manually ATM. I think I pretty much explained the pros and cons of switching to horizontal bar in the comment I linked you to. We may add Tag Cloud to Lens before serverless, so if you feel strongly you can leave them as they are. Worst-case scenario is that on serverless, they are read-only. At the end of the day, this is your call.

It seems that the Compatibility palette is called "kibana_palette" in the JSON. Is it valid to just replace that name with "default" in the JSON directly? Digging through the UI to replace 20 instance of this seems sort of inefficient.

You can just delete the whole palette property.

"palette": {
      "type": "palette",
      "name": "kibana_palette"
},

The choice of staying with the legacy metric was because of the increased font size for this compared to the alternative. Is it possible to increase font size to use more/waste less of the available panel in the new metric visualisation?

We've received this feedback before and are making the font larger and the breakpoints more generous in 8.8. But, we don't give users control over the size to promote alignment between metrics. Whitespace in visualizations isn't always wasted—it can actually promote faster comprehension. See this discuss post for more of our design thinking and the problems the new design solves.

Again, the choice is yours, but we would really like to see the new metric adopted here. We are actively working on ironing out the imperfections in the new design to where the legacy metric can be deprecated and eventually removed.

efd6 · 2023-03-31T20:13:48Z

Whitespace in visualizations isn't always wasted—it can actually promote faster comprehension. See this discuss post for more of our design thinking and the problems the new design solves.

Agree that in context whitespace is useful. I don't think it adds here since it's not separating things that need separation. My concern is the size and the non-centrality (tight border adjacency reduces the helpful whitespace) of the indicator and how that impacts on DEI (due to visual impairment). I didn't see anything in that post that gave evidence for reduced cognitive load with the new design unless perhaps appealing to the non-conformity of size. Did I miss it?

drewdaemon · 2023-03-31T20:49:02Z

Thanks for the feedback.

My concern is the size and the non-centrality of the indicator and how that impacts on DEI (due to visual impairment).

I'm glad you're thinking this way. It does occur to me that most of the other textual data displayed in our visualizations are smaller than this (e.g. on a slice label, in a legend). We're also careful with contrast (we enforce WCAG AA compliance) which may mitigate some of the concern. However, I'll bring this up with the team since it could be an important area for improvement.

I didn't see anything in that post that gave evidence for reduced cognitive load with the new design unless perhaps appealing to the non-conformity of size. Did I miss it?

I think that the value of the new design is made clearer on the screenshots in that discussion. With the old metric, a) the visual alignment across multiple metrics is often broken and b) figures that actually have a smaller numeric value often appear larger, both sources of cognitive load.

If you want to discuss more or don't feel like I've fully understood your feedback, please feel free to reach out in #kibana-visualizations. I want to make sure we're hearing you.

As I said above, new investment will center on the metric visualization. However, I'm merely acting as a consultant so you have the final say here.

efd6 · 2023-04-02T22:38:35Z

Thanks @drewdaemon. I've removed the palette clauses, but I'll leave the metrics as they are until further discussion has been had.

drewdaemon · 2023-04-06T17:52:28Z

FWIW, the team just decided to add tag cloud to Lens some time this year. Maybe around 8.10.

narph · 2023-04-28T09:22:07Z

@drewdaemon , @elastic/security-external-integrations can we get some feedback/approval on this PR, it has been opened for a while?

cc @andrewkroh

drewdaemon

Apologies. I thought I had already approved!

No "Convert to Lens" option was available for "Logon Sources [Windows System Security]".

Word cloud, "Target Users [Windows System Security]", not converted.

Did not convert "Group Management Events - Target Groups - Tag Cloud [Windows System Security]". In the "Group Management Events - Groups vs Actions - Heatmap [Windows System Security]" the horizontal axis is visually broken because labels may not be rotated or omitted.

Note that in the case of the user events, this appears to result in significant semantic differences.

efd6 · 2023-04-30T22:47:28Z

@rdner Can you approve please?

elasticmachine · 2023-05-02T08:35:53Z

Package system - 1.27.0 containing this change is available at https://epr.elastic.co/search?package=system

efd6 added enhancement New feature or request Integration:system System dashboard Relates to a Kibana dashboard bug, enhancement, or modification. labels Mar 30, 2023

efd6 requested a review from a team March 30, 2023 01:44

efd6 self-assigned this Mar 30, 2023

efd6 force-pushed the 5022-lens branch from 75d41fb to b9de727 Compare March 30, 2023 04:21

efd6 marked this pull request as ready for review March 30, 2023 04:42

efd6 requested review from a team as code owners March 30, 2023 04:42

efd6 requested review from rdner and cmacknz March 30, 2023 04:42

drewdaemon reviewed Mar 30, 2023

View reviewed changes

narph requested a review from drewdaemon April 28, 2023 09:17

drewdaemon approved these changes Apr 28, 2023

View reviewed changes

andrewkroh approved these changes Apr 28, 2023

View reviewed changes

efd6 added 3 commits May 1, 2023 07:19

convert overview to lens

04e69a8

convert user logon information to lens

c533fab

No "Convert to Lens" option was available for "Logon Sources [Windows System Security]".

convert failed and account lockout to lens

4092cba

efd6 added 7 commits May 1, 2023 07:19

convert user management events to lens

1568faa

Word cloud, "Target Users [Windows System Security]", not converted.

update changelog and manifest

547ba09

address pr comment: remove reference to TSVB

5ef9f3b

address pr comment: re-write tag clouds as bar charts

8bf6071

Note that in the case of the user events, this appears to result in significant semantic differences.

address pr comment: remove palette clauses

1051507

remove "(converted)" suffix

ac588ec

efd6 force-pushed the 5022-lens branch from e14ee6b to ac588ec Compare April 30, 2023 21:50

rdner approved these changes May 2, 2023

View reviewed changes

efd6 merged commit de439ef into elastic:main May 2, 2023

This was referenced May 4, 2023

Error message is displayed to user, when user is trying to add AWS integrations to agent policies. #6088

Closed

Duplicate tags entries are created on Stack Management> Tags page, when user creates a new policy in Fleet. #6096

Closed

efd6 deleted the 5022-lens branch February 5, 2025 22:05

system: convert visualisations to lens #5740

system: convert visualisations to lens #5740

Uh oh!

Conversation

efd6 commented Mar 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Overview

User Logon Information

Logon Failed and Account Lockout

fold

User Management Events

fold

Group Management Events

fold

Uh oh!

elasticmachine commented Mar 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Mar 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

rdner commented Mar 30, 2023

Uh oh!

efd6 commented Mar 30, 2023

Uh oh!

cmacknz commented Mar 30, 2023

Uh oh!

drewdaemon left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 commented Mar 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drewdaemon commented Mar 31, 2023

Uh oh!

efd6 commented Mar 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drewdaemon commented Mar 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efd6 commented Apr 2, 2023

Uh oh!

drewdaemon commented Apr 6, 2023

Uh oh!

narph commented Apr 28, 2023

Uh oh!

drewdaemon left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 commented Apr 30, 2023

Uh oh!

elasticmachine commented May 2, 2023

Uh oh!

Uh oh!

efd6 commented Mar 30, 2023 •

edited

Loading

elasticmachine commented Mar 30, 2023 •

edited

Loading

elasticmachine commented Mar 30, 2023 •

edited

Loading

efd6 commented Mar 31, 2023 •

edited

Loading

efd6 commented Mar 31, 2023 •

edited

Loading

drewdaemon commented Mar 31, 2023 •

edited

Loading