Skip to content

system: convert dashboard visualisations to store by value #5322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 14, 2023

Conversation

efd6
Copy link
Contributor

@efd6 efd6 commented Feb 20, 2023

What does this PR do?

This updates visualisations to store by value.

The operations used are shown in the git-generate script in the commit message and can be confirmed (slowly) by running git generate.

Output of script from point of cloning the conversion tool.

Cloning into 'legacy_vis_analyzer'...
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 12 (delta 0), reused 5 (delta 0), pack-reused 0
Receiving objects: 100% (12/12), 11.94 KiB | 643.00 KiB/s, done.
Submodule 'beats' (https://github.com/elastic/beats) registered for path 'beats'
Submodule 'integrations' (https://github.com/elastic/integrations) registered for path 'integrations'
Submodule 'kibana' (https://github.com/elastic/kibana) registered for path 'kibana'
Cloning into '/Users/daniel/go/src/github.com/elastic/integrations/packages/system/legacy_vis_analyzer/beats'...
Cloning into '/Users/daniel/go/src/github.com/elastic/integrations/packages/system/legacy_vis_analyzer/integrations'...
Cloning into '/Users/daniel/go/src/github.com/elastic/integrations/packages/system/legacy_vis_analyzer/kibana'...
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote: Enumerating objects: 2269, done.
remote: Counting objects: 100% (2269/2269), done.
remote: Compressing objects: 100% (1100/1100), done.
remote: Total 1366 (delta 893), reused 494 (delta 210), pack-reused 0
Receiving objects: 100% (1366/1366), 2.09 MiB | 2.80 MiB/s, done.
Resolving deltas: 100% (893/893), completed with 614 local objects.
From https://github.com/elastic/beats
 * branch              b83248684299976dd4630064d6c1f7413e945f3b -> FETCH_HEAD
Submodule path 'beats': checked out 'b83248684299976dd4630064d6c1f7413e945f3b'
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote: Enumerating objects: 11640, done.
remote: Counting objects: 100% (11640/11640), done.
remote: Compressing objects: 100% (4507/4507), done.
remote: Total 9089 (delta 4407), reused 6437 (delta 2905), pack-reused 0
Receiving objects: 100% (9089/9089), 11.91 MiB | 3.00 MiB/s, done.
Resolving deltas: 100% (4407/4407), completed with 724 local objects.
From https://github.com/elastic/integrations
 * branch              a3639be8de8b6f564644c6a9afc93170a2b915ef -> FETCH_HEAD
Submodule path 'integrations': checked out 'a3639be8de8b6f564644c6a9afc93170a2b915ef'
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote: Enumerating objects: 33054, done.
remote: Counting objects: 100% (33054/33054), done.
remote: Compressing objects: 100% (18606/18606), done.
remote: Total 22258 (delta 12214), reused 9539 (delta 3140), pack-reused 0
Receiving objects: 100% (22258/22258), 28.54 MiB | 2.20 MiB/s, done.
Resolving deltas: 100% (12214/12214), completed with 7951 local objects.
From https://github.com/elastic/kibana
 * branch                7a3243b79fbc309875490e1477a3d97b80854676 -> FETCH_HEAD
Submodule path 'kibana': checked out '7a3243b79fbc309875490e1477a3d97b80854676'
yarn install v1.22.19
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 🔨  Building fresh packages...
✨  Done in 0.64s.
(node:37532) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
(Use `node --trace-warnings ...` to show where the warning was created)
Prepared 45 visualizations to be inlined
Processing dashboard [System Windows Security] Group Management Events - Simple Metrics
Inlined a vis, pushed 0 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Leaving panel of type search
Leaving panel of type search
Inlined a vis, pushed 0 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [System Windows Security] User Logons - Simple Metrics
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 0 inner references
Leaving panel of type search
Inlined a vis, pushed 0 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 1 inner references
Leaving panel of type search
Inlined a vis, pushed 2 inner references
Leaving panel of type search
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [Logs System] New users and groups
Inlined a vis, pushed 0 inner references
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [Logs System] Sudo commands
Inlined a vis, pushed 0 inner references
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [Logs System] SSH login attempts
Leaving panel of type search
Inlined a vis, pushed 0 inner references
Leaving panel of type map, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [System Windows Security] User Management Events
Inlined a vis, pushed 0 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Leaving panel of type search
Leaving panel of type search
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 0 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [Metrics System] Host overview
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type lens, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type lens, seems to be inlined already
Leaving panel of type lens, seems to be inlined already
Leaving panel of type lens, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type lens, seems to be inlined already
Leaving panel of type lens, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type lens, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type lens, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [System Windows Security] User Management Events - Simple Metric
Inlined a vis, pushed 0 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Leaving panel of type search
Leaving panel of type search
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 0 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [Logs System] Syslog dashboard
Leaving panel of type search
Inlined a vis, pushed 0 inner references
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [Metrics System] Overview
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type lens, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type lens, seems to be inlined already
Leaving panel of type lens, seems to be inlined already
Processing dashboard [System] Windows Overview
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [System Windows Security] User Logons
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 0 inner references
Leaving panel of type search
Inlined a vis, pushed 0 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 1 inner references
Leaving panel of type search
Inlined a vis, pushed 2 inner references
Leaving panel of type search
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [System Windows Security] Group Management Events
Inlined a vis, pushed 0 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Leaving panel of type search
Leaving panel of type search
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 1 inner references
Inlined a vis, pushed 0 inner references
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [System Windows Security] Failed and Blocked Accounts
Inlined a vis, pushed 0 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 3 inner references
Leaving panel of type search
Inlined a vis, pushed 3 inner references
Inlined a vis, pushed 3 inner references
Inlined a vis, pushed 0 inner references
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Processing dashboard [System Windows Security] Failed and Blocked Accounts - Simple Metrics
Inlined a vis, pushed 0 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 2 inner references
Inlined a vis, pushed 3 inner references
Leaving panel of type search
Inlined a vis, pushed 3 inner references
Inlined a vis, pushed 3 inner references
Inlined a vis, pushed 0 inner references
Leaving panel of type visualization, seems to be inlined already
Leaving panel of type visualization, seems to be inlined already
Inlined 44 visualizations
Inlined 0 maps
Inlined 0 lenses
Some visualizations did not get inlined! 44/45
Did not inline system-f42f3b20-fee6-11e9-8405-516218e3d268 anywhere
Removing visualization folder
Writing back dashboards

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Install integrations and examine dashboards.

Related issues

Screenshots

@efd6 efd6 requested a review from a team February 20, 2023 02:44
@efd6 efd6 self-assigned this Feb 20, 2023
@elasticmachine
Copy link

elasticmachine commented Feb 20, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-02-27T04:22:36.208+0000

  • Duration: 15 min 6 sec

Test stats 🧪

Test Results
Failed 0
Passed 149
Skipped 0
Total 149

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Feb 20, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (3/3) 💚
Files 100.0% (4/4) 💚 2.206
Classes 100.0% (4/4) 💚 2.206
Methods 60.759% (48/79) 👎 -30.565
Lines 99.535% (2780/2793) 👍 6.566
Conditionals 100.0% (0/0) 💚

@efd6 efd6 marked this pull request as ready for review February 20, 2023 03:37
@efd6 efd6 requested review from a team as code owners February 20, 2023 03:37
@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

The git-generate script requires yarn to be installed.

[git-generate]

cd packages/system/
elastic-package stack down && elastic-package stack up --version=8.1.0 -d -v
git clone --depth=1 https://github.com/flash1293/legacy_vis_analyzer.git
(
	cd legacy_vis_analyzer
	git submodule update --depth=1 --init --recursive
	yarn
	KIBANA="https://elastic:changeme@localhost:5601" NODE_TLS_REJECT_UNAUTHORIZED='0' node inline.js ../kibana/
)
rm -rf legacy_vis_analyzer
gsed -i -r 's/^version: 1\.[0-9]+\.[0-9]+$/version: 1.25.0/' manifest.yml
gsed -i '/^# newer versions go on top$/d' changelog.yml
echo '# newer versions go on top' >changelog.yml~
cat <<EOF | cat - changelog.yml >>changelog.yml~ && mv changelog.yml{~,}
- version: "1.25.0"
  changes:
    - description: Convert dashboard visualisations to storage by value.
      type: enhancement
      link: elastic#5322
EOF
@efd6 efd6 marked this pull request as draft March 2, 2023 00:11
Copy link
Contributor

@drewdaemon drewdaemon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work. Just for visibility, there is automated tooling to accomplish this.

@efd6
Copy link
Contributor Author

efd6 commented Mar 13, 2023

@drewdaemon That package is the same as the one that I used here, correct?

@efd6 efd6 marked this pull request as ready for review March 13, 2023 23:14
@efd6 efd6 merged commit 3517e64 into elastic:main Mar 14, 2023
@elasticmachine
Copy link

Package system - 1.25.0 containing this change is available at https://epr.elastic.co/search?package=system

@drewdaemon
Copy link
Contributor

@efd6 great! There's no way for me to know if you used the tool or did it in the dashboard UI, so I wanted to make sure you knew.

@efd6
Copy link
Contributor Author

efd6 commented Mar 15, 2023

@drewdaemon The commit message has a git-generate script which re-runs the process and shows what was done (excellent tool).

agithomas pushed a commit to agithomas/integrations that referenced this pull request Mar 20, 2023
)

The git-generate script requires yarn to be installed.

[git-generate]

cd packages/system/
elastic-package stack down && elastic-package stack up --version=8.1.0 -d -v
git clone --depth=1 https://github.com/flash1293/legacy_vis_analyzer.git
(
	cd legacy_vis_analyzer
	git submodule update --depth=1 --init --recursive
	yarn
	KIBANA="https://elastic:changeme@localhost:5601" NODE_TLS_REJECT_UNAUTHORIZED='0' node inline.js ../kibana/
)
rm -rf legacy_vis_analyzer
gsed -i -r 's/^version: 1\.[0-9]+\.[0-9]+$/version: 1.25.0/' manifest.yml
gsed -i '/^# newer versions go on top$/d' changelog.yml
echo '# newer versions go on top' >changelog.yml~
cat <<EOF | cat - changelog.yml >>changelog.yml~ && mv changelog.yml{~,}
- version: "1.25.0"
  changes:
    - description: Convert dashboard visualisations to storage by value.
      type: enhancement
      link: elastic#5322
EOF
agithomas pushed a commit to agithomas/integrations that referenced this pull request Mar 21, 2023
)

The git-generate script requires yarn to be installed.

[git-generate]

cd packages/system/
elastic-package stack down && elastic-package stack up --version=8.1.0 -d -v
git clone --depth=1 https://github.com/flash1293/legacy_vis_analyzer.git
(
	cd legacy_vis_analyzer
	git submodule update --depth=1 --init --recursive
	yarn
	KIBANA="https://elastic:changeme@localhost:5601" NODE_TLS_REJECT_UNAUTHORIZED='0' node inline.js ../kibana/
)
rm -rf legacy_vis_analyzer
gsed -i -r 's/^version: 1\.[0-9]+\.[0-9]+$/version: 1.25.0/' manifest.yml
gsed -i '/^# newer versions go on top$/d' changelog.yml
echo '# newer versions go on top' >changelog.yml~
cat <<EOF | cat - changelog.yml >>changelog.yml~ && mv changelog.yml{~,}
- version: "1.25.0"
  changes:
    - description: Convert dashboard visualisations to storage by value.
      type: enhancement
      link: elastic#5322
EOF
@efd6 efd6 deleted the 5022-by-value branch February 5, 2025 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Integration:system System
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants