[Prometheus] Add dynamic mapping to fix prometheus histogram type

[crespocarlos]

What does this PR do?

This PR fixes a problem that the Prometheus package has to ingest histogram data type

I've tried different approaches but this one seems to be the only one that works.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

• Install Prometheus or use a docker container
  a. I've used a docker image following this doc and this example for the prometheus.yml config file
  b. Make sure that http://localhost:9090/metrics has data
• Install the Prometheus integration
  a. If using docker, run docker run -it --rm alpine nslookup host.docker.internal to retrieve the host
  b. DISABLE the Leader Election option (it seems that we have a bug here too)
  c. Clear all authentication fields

Related issues

Closes #2257

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-07-29T12:31:49.330+0000

• Duration: 15 min 47 sec

Test stats 🧪

Test	Results
Failed	0
Passed	15
Skipped	0
Total	15

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚 2.868
Classes	100.0% (0/0)	💚 2.868
Methods	66.667% (6/9)	👎 -22.584
Lines	100.0% (0/0)	💚 9.399
Conditionals	100.0% (0/0)	💚

[tetianakravchenko]

what is the type of this field before adding this configuration?
according to https://github.com/elastic/integrations/blob/main/packages/prometheus/data_stream/collector/fields/fields.yml#L38, it should be histogram. I will try to reproduce it to verify, maybe there is some mistake in field definition.

[crespocarlos]

that doesn't work for some reason. I think it considers type: object and not object_type: histogram. I've tried changing type to histogram, but it didn't work

[crespocarlos]

it's actually more related to this: elastic/kibana#129344

[tetianakravchenko]

I think it considers type: object and not object_type: histogram

I've checked it, with previous version mapping looks like:

[tetianakravchenko]

As well dynamic_templates should be added for remote_write data_stream and checked if it should be adjusted for other fields like prometheus.*.value, etc. Added it as comment for existing issue - #3668 (comment)

[tetianakravchenko]

b. DISABLE the Leader Election option (it seems that we have a bug here too)

created the issue - #3905

[matschaffer]

In trying to test prometheus histogrms today I got this error:

Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.August, 18, 15, 50, 35, 391910000, time.Local), Meta:{"raw_index":"metrics-prometheus.collector-default"}, Fields:{"agent":{"ephemeral_id":"1737dd84-fa6c-42e0-a988-b2cf15b463fb","id":"e2ca571c-fa82-47f6-ae5d-d207e229e434","name":"matschaffer-mbp2019.lan","type":"metricbeat","version":"8.3.3"},"data_stream":{"dataset":"prometheus.collector","namespace":"default","type":"metrics"},"ecs":{"version":"8.0.0"},"elastic_agent":{"id":"e2ca571c-fa82-47f6-ae5d-d207e229e434","snapshot":false,"version":"8.3.3"},"event":{"dataset":"prometheus.collector","duration":20688852,"module":"prometheus"},"host":{"architecture":"x86_64","hostname":"matschaffer-mbp2019.lan","id":"815EB661-322B-5B8B-A0BA-C83E911AC99A","ip":["fe80::aede:48ff:fe00:1122","fe80::5e:dea2:54ee:3d16","192.168.86.31","240b:11:6401:6f0:1c5e:41e4:4952:182f","240b:11:6401:6f0:dd4e:33ef:f2d9:d984","fe80::ec97:dff:fed5:4eb5","fe80::ec97:dff:fed5:4eb5","fe80::6d53:32bb:12ba:b7f7","fe80::2f1f:ad6a:53f3:e6d1","fe80::ce81:b1c:bd2c:69e","fe80::c76c:36b6:3eaf:bf39","fe80::1ac9:3572:ad1b:8c44","fe80::e8b5:e977:a471:ab5e","fe80::cf63:dfe0:58f6:5c33","fe80::bbfc:21ab:a03e:1beb","fe80::67b2:f8ec:fefd:8a2e"],"mac":["3c:22:fb:a2:28:5f","3e:22:fb:a2:28:5f","82:c1:12:47:1c:00","82:c1:12:47:1c:01","82:c1:12:47:1c:04","82:c1:12:47:1c:05","ac:de:48:00:11:22","ee:97:0d:d5:4e:b5"],"name":"matschaffer-mbp2019.lan","os":{"build":"21G72","family":"darwin","kernel":"21.6.0","name":"macOS","platform":"darwin","type":"macos","version":"12.5"}},"metricset":{"name":"collector","period":10000},"prometheus":{"labels":{"instance":"localhost:5601","job":"prometheus","ruleType":"monitoring_alert_nodes_changed"},"ruleDurationsByType":{"histogram":{"counts":[0,0,0,0,0,0,0,0,0,0,0],"values":[0,2.5,7.5,17.5,37.5,62.5,87.5,175,375,750,1500]}},"ruleExecutions_total":{"counter":34,"rate":0}},"service":{"address":"http://localhost:5601/ftw/api/monitoring_collection/v1/prometheus","type":"prometheus"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"mapper [prometheus.ruleDurationsByType.histogram.values] cannot be changed from type [long] to [float]"}, dropping event!
--

But the index in question already has the field as a histogram with this change in place which is strange:

GET .ds-metrics-prometheus.collector-default-2022.08.08-000003/_mapping
...
      "dynamic_templates": [
        {
          "histogram": {
            "path_match": "prometheus.*.histogram",
            "mapping": {
              "type": "histogram"
            }
          }
        },
...
            "ruleDurationsByType": {
              "properties": {
                "histogram": {
                  "type": "histogram"
                }
              }
            },
...

I'll try nuking the datastream incase that helps.

[matschaffer]

Delete and disabling the leader election flag seemed to get things moving. Thanks for the above info!