[oracle] Enhancement to capture system statistics metrics, pga metrics, sga metrics #3967

agithomas · 2022-08-08T06:55:46Z

Enhancement to capture system statistics metrics, pga metrics, sga metrics.

Type of change

Enhancement

What does this PR do?

Enhancement to capture system statistics metrics, pga metrics, sga metrics.
Added dashboard to capture performance metrics, system statistics metrics, pga and sga metrics.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

docker run -d --name oracle -p 1521:1521 -p 5500:5500 docker.elastic.co/observability-ci/database-enterprise:12.2.0.1
Install oracle client in the host / vm
elastic-package stack up --version 8.3.0
Access kibana, login, configure oracle integration. Use the default values

Related issues

Relates Oracle Packages Enahcements #3911

Screenshots

…trics. Added dashboard to capture performance metrics, system statistics metrics, pga and sga metrics.

elasticmachine · 2022-08-08T07:10:58Z

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-09-12T08:54:33.628+0000
Duration: 24 min 39 sec

Steps errors

Expand to view the steps failures

`Google Storage Download`

Took 0 min 0 sec . View more details here

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-08-08T07:11:09Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚 2.994
Classes	100.0% (`1/1`)	💚 2.994
Methods	82.759% (`24/29`)	👎 -7.024
Lines	89.13% (`164/184`)	👎 -2.392
Conditionals	100.0% (`0/0`)	💚

packages/oracle/changelog.yml

packages/oracle/data_stream/performance/fields/fields.yml

packages/oracle/data_stream/system_statistics/fields/fields.yml

ishleenk17

ECS and base fields not present in performance data stream

ishleenk17

IIRC, SQL has flexibility of letting the user choose which field they want to retrive from a table in DB.
Would that be inherited by oracle as well since its using sql metrics ?

agithomas · 2022-09-07T05:10:25Z

IIRC, SQL has flexibility of letting the user choose which field they want to retrive from a table in DB. Would that be inherited by oracle as well since its using sql metrics ?

dynamic_metric_name_filter feature is present to permit the user to configure based on his / her requirements

agithomas · 2022-09-07T05:46:37Z

ECS and base fields not present in performance data stream

It is present. But, you won't find in the Files Changed. We already have Performance data stream and this time we are extending it. base-fields.yml and ecs.yml files were added earlier.

agithomas · 2022-09-12T07:05:07Z

@efd6 , all the issues highlighted are fixed. Kindly review

… feature-3911

elasticmachine · 2022-09-12T09:18:29Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-09-20T12:36:49.372+0000
Duration: 13 min 23 sec

Test stats 🧪

Test	Results
Failed	0
Passed	21
Skipped	0
Total	21

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

efd6

After nit this LGTM. The oracle SQL should probably also be checked by another; it looks OK to this non-expert.

packages/oracle/changelog.yml

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

packages/oracle/data_stream/performance/fields/fields.yml

packages/oracle/data_stream/system_statistics/fields/fields.yml

muthu-mps · 2022-09-14T04:58:09Z

Can we realign the dashboard packages/oracle/img/Oracle-system_statistics-dashboard.png and remove the extra gap after Inactive session count > 1hr.

packages/oracle/data_stream/system_statistics/fields/fields.yml

packages/oracle/data_stream/pga_sga/fields/fields.yml

… feature-3911

agithomas · 2022-09-15T12:21:36Z

Can we realign the dashboard packages/oracle/img/Oracle-system_statistics-dashboard.png and remove the extra gap after Inactive session count > 1hr.

Corrected.

lalit-satapathy · 2022-09-15T12:11:01Z

packages/oracle/data_stream/performance/agent/stream/stream.yml.hbs

+    response_format: table
+
+


How many documents are created by these queries? Should we add request_merge flag? This question applies to all the oracle data streams.

pga_sga : 2
sysmetric : 1
system_statistics : 1

There is a scope of improvement in performance datastream (part of 8.4) which has 7 queries.

As per the discussion, a separate PR will be created for this as 8.4.0 features are needed, changing min kibana version to 8.4.0 for this integration

lalit-satapathy · 2022-09-15T12:16:41Z

packages/oracle/data_stream/performance/fields/fields.yml

+          unit: percent
+          metric_type: gauge
+          description: Percentage of time waits that are not Idle wait class.
+        - name: pct_waits


should we remove "wait" for last field names. Need to think if having these in the names better since the group name is "wait"?

- name: pct_time type: double unit: percent metric_type: gauge description: Percentage of time waits that are not Idle wait class. - name: pct_waits type: double unit: percent metric_type: gauge description: Percentage of number of pct time waits that are not of Idle wait class.

Cannot remove wait because, it will lead to conflicts / confusions

lalit-satapathy · 2022-09-15T12:27:58Z

packages/oracle/data_stream/performance/elasticsearch/ingest_pipeline/default.yml

+        ignore_failure: true
+    - rename:
+        field: oracle.performance.pct_time
+        target_field: oracle.performance.wait.pct_time


Can we make sure these oracle.performance.wait.* fields are there in the sample events? also I am not able to locate them in the README file also.

packages/oracle/data_stream/performance/fields/fields.yml

packages/oracle/data_stream/system_statistics/fields/fields.yml

packages/oracle/data_stream/pga_sga/fields/fields.yml

packages/oracle/_dev/build/docs/README.md

elasticmachine · 2022-09-19T05:52:59Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/oracle/_dev/build/docs/README.md

packages/oracle/data_stream/performance/agent/stream/stream.yml.hbs

agithomas · 2022-09-20T07:08:55Z

@lalit-satapathy , @muthu-mps , @efd6 ,

Can you please help reviewing the recent change and approve the PR?

From your last review, below changes are made

Modified datastream name from pga_sga to memory
Field mapping in datastream memory modified
Modified field name having percentage to pct
Added screenshots of memory metrics dashboard
README file wasn't displaying deep nested fields. This bug is fixed.
Typos fixed.

packages/oracle/_dev/build/docs/README.md

muthu-mps · 2022-09-20T11:53:07Z

packages/oracle/manifest.yml

@@ -19,7 +48,7 @@ icons:
 policy_templates:
  - name: oracle
    title: Oracle Audit Logs
-    description: Collect Oracle Audit logs, Performance metrics, Tablespace metrics, Sysmetrics integration
+    description: Collect Oracle Audit logs, Performance metrics, Tablespace metrics, Sysmetrics integration, System statistics, memory metrics.


memory metric M. Can we make this to camel case ?

muthu-mps · 2022-09-20T11:57:53Z

packages/oracle/data_stream/memory/elasticsearch/ingest_pipeline/default.yml

+    - rename:
+        field: oracle.metrics
+        target_field: oracle.memory
+        ignore_missing: true


Do you consider adding ignore_failure for rest of the fields ? Because the other datastreams has ignore_failure for all the fields.

muthu-mps

LGTM!

Enhancement to capture system statistics metrics, pga metrics, sga me…

3359cdb

…trics. Added dashboard to capture performance metrics, system statistics metrics, pga and sga metrics.

Added dashboard images, fixed sample files

6116731

agithomas mentioned this pull request Aug 8, 2022

Oracle Packages Enahcements #3911

Closed

andrewkroh changed the title ~~Enhancement to capture system statistics metrics, pga metrics, sga metrics~~ [oracle] Enhancement to capture system statistics metrics, pga metrics, sga metrics Aug 14, 2022

andrewkroh added Integration:oracle Oracle enhancement New feature or request labels Aug 14, 2022

agithomas marked this pull request as ready for review August 29, 2022 04:51

agithomas requested a review from a team as a code owner August 29, 2022 04:51

agithomas self-assigned this Aug 29, 2022

efd6 reviewed Aug 29, 2022

View reviewed changes

packages/oracle/changelog.yml Outdated Show resolved Hide resolved

packages/oracle/data_stream/performance/fields/fields.yml Outdated Show resolved Hide resolved

ishleenk17 reviewed Sep 6, 2022

View reviewed changes

packages/oracle/data_stream/system_statistics/fields/fields.yml Show resolved Hide resolved

ishleenk17 reviewed Sep 6, 2022

View reviewed changes

Fixes based on review comments

8f25af3

agithomas requested review from lalit-satapathy and muthu-mps September 12, 2022 07:04

agithomas added 2 commits September 12, 2022 09:01

Changes made to support the field_name changes

e600817

Merge branch 'feature-3911' of github.com:agithomas/integrations into…

f731214

… feature-3911

Fixed the issue number

f65f54e

efd6 reviewed Sep 12, 2022

View reviewed changes

packages/oracle/changelog.yml Outdated Show resolved Hide resolved

Update packages/oracle/changelog.yml

42bbf49

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

muthu-mps reviewed Sep 14, 2022

View reviewed changes

muthu-mps reviewed Sep 15, 2022

View reviewed changes

packages/oracle/data_stream/system_statistics/fields/fields.yml Outdated Show resolved Hide resolved

packages/oracle/data_stream/pga_sga/fields/fields.yml Outdated Show resolved Hide resolved

packages/oracle/data_stream/pga_sga/fields/fields.yml Outdated Show resolved Hide resolved

agithomas added 4 commits September 15, 2022 05:28

Correct field mapping description

938baed

Merge branch 'feature-3911' of github.com:agithomas/integrations into…

bd7c267

… feature-3911

Made adjustments to field names

846b858

Modified field names

1ebb3b6

lalit-satapathy reviewed Sep 15, 2022

View reviewed changes

Static testing fixes

f3f7963

muthu-mps reviewed Sep 15, 2022

View reviewed changes

packages/oracle/data_stream/performance/fields/fields.yml Outdated Show resolved Hide resolved

packages/oracle/data_stream/performance/fields/fields.yml Outdated Show resolved Hide resolved

muthu-mps reviewed Sep 15, 2022

View reviewed changes

packages/oracle/data_stream/system_statistics/fields/fields.yml Outdated Show resolved Hide resolved

packages/oracle/data_stream/pga_sga/fields/fields.yml Outdated Show resolved Hide resolved

muthu-mps reviewed Sep 15, 2022

View reviewed changes

packages/oracle/_dev/build/docs/README.md Outdated Show resolved Hide resolved

Field mapping changes made

4ef6f57

agithomas added 5 commits September 19, 2022 09:08

Fixed README file issues

f45a2b4

Changed datastream name from pga_stat to memory

6a2c2c6

Renamed pga-sga to memory. Added screenshot

68c2d20

Modified dashboard to support field name changes

d273d13

Added new dashboard screenshot

54b4357

muthu-mps reviewed Sep 20, 2022

View reviewed changes

packages/oracle/_dev/build/docs/README.md Outdated Show resolved Hide resolved

packages/oracle/data_stream/performance/agent/stream/stream.yml.hbs Outdated Show resolved Hide resolved

agithomas added 2 commits September 20, 2022 06:16

Fixed dashboard compatibility issue, typo issues

9bb4a3a

Removed empty line

c9f4116

muthu-mps reviewed Sep 20, 2022

View reviewed changes

Added additional documentation

d17cee2

muthu-mps self-requested a review September 20, 2022 13:00

muthu-mps approved these changes Sep 20, 2022

View reviewed changes

efd6 approved these changes Sep 21, 2022

View reviewed changes

agithomas merged commit 6b46ac9 into elastic:main Sep 21, 2022

andrewkroh mentioned this pull request Sep 29, 2022

[production] Promote many packages from snapshot elastic/package-storage#5760

Merged

agithomas deleted the feature-3911 branch February 12, 2025 05:02

andrewkroh added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Mar 13, 2025

[oracle] Enhancement to capture system statistics metrics, pga metrics, sga metrics #3967

[oracle] Enhancement to capture system statistics metrics, pga metrics, sga metrics #3967

Uh oh!

Conversation

agithomas commented Aug 8, 2022

Type of change

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Aug 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❕ Build Aborted

Build stats

Steps errors

Google Storage Download

🤖 GitHub comments

Uh oh!

elasticmachine commented Aug 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ishleenk17 left a comment

Choose a reason for hiding this comment

Uh oh!

ishleenk17 left a comment

Choose a reason for hiding this comment

Uh oh!

agithomas commented Sep 7, 2022

Uh oh!

agithomas commented Sep 7, 2022

Uh oh!

agithomas commented Sep 12, 2022

Uh oh!

elasticmachine commented Sep 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

muthu-mps commented Sep 14, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

agithomas commented Sep 15, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Sep 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

elasticmachine commented Aug 8, 2022 •

edited

Loading

`Google Storage Download`

elasticmachine commented Aug 8, 2022 •

edited

Loading

elasticmachine commented Sep 12, 2022 •

edited

Loading

elasticmachine commented Sep 19, 2022 •

edited

Loading