Skip to content

[AWS] [EMR] Add logs data stream #6895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Jul 20, 2023
Merged

[AWS] [EMR] Add logs data stream #6895

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d0374fd
add emr logs data stream
gpop63 Jul 9, 2023
cf05a92
fix sample_event
gpop63 Jul 17, 2023
694dbdd
make pipeline log classpath shorter
gpop63 Jul 17, 2023
15a022b
make s3 enabled and cloudwatch disabled by default
gpop63 Jul 17, 2023
26fb0a1
replace EC2 with EMR in manifest
gpop63 Jul 17, 2023
52f6363
Merge remote-tracking branch 'upstream/main' into add_aws-emr_logs
gpop63 Jul 17, 2023
139eb41
fix sample_event
gpop63 Jul 17, 2023
633db99
add aws s3 fields
gpop63 Jul 18, 2023
415ca48
add aws s3 standalone options
gpop63 Jul 18, 2023
e6cc04f
Merge remote-tracking branch 'upstream/main' into add_aws-emr_logs
gpop63 Jul 18, 2023
d90a939
Merge branch 'main' into add_aws-emr_logs
gpop63 Jul 19, 2023
9ee9322
add logs in docs
gpop63 Jul 19, 2023
a257e29
Merge remote-tracking branch 'upstream/main' into add_aws-emr_logs
gpop63 Jul 19, 2023
aa565fa
Merge remote-tracking branch 'upstream/main' into add_aws-emr_logs
gpop63 Jul 20, 2023
e7caaf7
fix config order
gpop63 Jul 20, 2023
3800ea2
remove redundant field
gpop63 Jul 20, 2023
cc56bf7
better fields naming
gpop63 Jul 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions packages/aws/_dev/build/docs/emr.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,14 @@ For example, you could use this data to track Amazon EMR cluster progress and cl

## Data streams

The Amazon EMR integration collects one type of data: metrics.
The Amazon EMR integration collects two types of data: metrics and logs.

**Metrics** give you insight into the state of Amazon EMR.
The metrics collected by the Amazon EMR integration include cluster progress, cluster state, cluster or node storage, and more. See more details in the [Metrics reference](#metrics-reference)

**Logs** help you keep a record of events happening in Amazon EMR.
Logs collected by the Amazon EMR integration include the cluster status, node status details and more.

## Requirements

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
Expand All @@ -41,4 +44,10 @@ For step-by-step instructions on how to set up an integration, see the

{{event "emr_metrics"}}

{{fields "emr_metrics"}}
{{fields "emr_metrics"}}

## Logs reference

{{event "emr_logs"}}

{{fields "emr_logs"}}
5 changes: 5 additions & 0 deletions packages/aws/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.50.0"
changes:
- description: Add EMR logs data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/6895
- version: "1.49.0"
changes:
- description: Add API Gateway logs datastream
Expand Down
7 changes: 7 additions & 0 deletions packages/aws/data_stream/emr_logs/_dev/test/pipeline/test-common-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
fields:
tags:
- preserve_original_event
multiline:
first_line_pattern: '^[0-9]'
negate: true
match: after
92 changes: 92 additions & 0 deletions packages/aws/data_stream/emr_logs/_dev/test/pipeline/test-emr-hadoop.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
2023-06-26 13:45:49,685 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ip-172-31-25-102.eu-central-1.compute.internal/172.31.25.102
STARTUP_MSG: args = [-format, -nonInteractive]
STARTUP_MSG: version = 3.3.3-amzn-3
STARTUP_MSG: classpath = /etc/hadoop/conf:/usr/lib/hadoop/lib/jetty-security-9.4.48.v20220622.jar:/usr/lib/hadoop/lib/accessors-smart-2.4.7.jar:/usr/lib/hadoop/lib/jersey-core-1.19.jar:/usr/lib/hadoop/lib/animal-sniffer-annotations-1.17.jar
STARTUP_MSG: build = Unknown -r Unknown; compiled by 'release' on 2023-05-31T03:49Z
STARTUP_MSG: java = 1.8.0_372
************************************************************/
2023-06-26 13:45:49,697 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2023-06-26 13:45:49,823 INFO namenode.NameNode: createNameNode [-format, -nonInteractive]
2023-06-26 13:45:50,318 INFO common.Util: Assuming 'file' scheme for path /mnt/namenode in configuration.
2023-06-26 13:45:50,319 INFO common.Util: Assuming 'file' scheme for path /mnt1/namenode in configuration.
2023-06-26 13:45:50,319 INFO common.Util: Assuming 'file' scheme for path /mnt/namenode in configuration.
2023-06-26 13:45:50,319 INFO common.Util: Assuming 'file' scheme for path /mnt1/namenode in configuration.
2023-06-26 13:45:50,330 INFO namenode.NameNode: Formatting using clusterid: CID-1b3b14b6-5518-47c3-b981-e5cb6b0ce38c
2023-06-26 13:45:50,394 INFO namenode.FSEditLog (main): Edit logging is async:true
2023-06-26 13:45:50,484 INFO namenode.FSNamesystem: KeyProvider: KeyProviderCryptoExtension: org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@a530d0a
2023-06-26 13:45:50,486 INFO namenode.FSNamesystem: fsLock is fair: true
2023-06-26 13:45:50,486 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2023-06-26 13:45:50,492 INFO namenode.FSNamesystem: fsOwner = hdfs (auth:SIMPLE)
2023-06-26 13:45:50,493 INFO namenode.FSNamesystem: supergroup = hdfsadmingroup
2023-06-26 13:45:50,493 INFO namenode.FSNamesystem: isPermissionEnabled = true
2023-06-26 13:45:50,493 INFO namenode.FSNamesystem: isStoragePolicyEnabled = true
2023-06-26 13:45:50,493 INFO namenode.FSNamesystem: HA Enabled: false
2023-06-26 13:45:50,566 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2023-06-26 13:45:50,601 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2023-06-26 13:45:50,601 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2023-06-26 13:45:50,610 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2023-06-26 13:45:50,611 INFO blockmanagement.BlockManager: The block deletion will start around 2023 Jun 26 13:45:50
2023-06-26 13:45:50,612 INFO util.GSet: Computing capacity for map BlocksMap
2023-06-26 13:45:50,613 INFO util.GSet: VM type = 64-bit
2023-06-26 13:45:50,614 INFO util.GSet: 2.0% max memory 864 MB = 17.3 MB
2023-06-26 13:45:50,614 INFO util.GSet: capacity = 2^21 = 2097152 entries
2023-06-26 13:45:50,623 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled
2023-06-26 13:45:50,623 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2023-06-26 13:45:50,631 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.999
2023-06-26 13:45:50,631 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2023-06-26 13:45:50,631 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 5000
2023-06-26 13:45:50,632 INFO blockmanagement.BlockManager: defaultReplication = 1
2023-06-26 13:45:50,632 INFO blockmanagement.BlockManager: maxReplication = 512
2023-06-26 13:45:50,632 INFO blockmanagement.BlockManager: minReplication = 1
2023-06-26 13:45:50,632 INFO blockmanagement.BlockManager: maxReplicationStreams = 100
2023-06-26 13:45:50,633 INFO blockmanagement.BlockManager: redundancyRecheckInterval = 3000ms
2023-06-26 13:45:50,633 INFO blockmanagement.BlockManager: encryptDataTransfer = false
2023-06-26 13:45:50,633 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
2023-06-26 13:45:50,670 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
2023-06-26 13:45:50,671 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
2023-06-26 13:45:50,671 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
2023-06-26 13:45:50,671 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
2023-06-26 13:45:50,696 INFO util.GSet: Computing capacity for map INodeMap
2023-06-26 13:45:50,696 INFO util.GSet: VM type = 64-bit
2023-06-26 13:45:50,697 INFO util.GSet: 1.0% max memory 864 MB = 8.6 MB
2023-06-26 13:45:50,697 INFO util.GSet: capacity = 2^20 = 1048576 entries
2023-06-26 13:45:50,699 INFO namenode.FSDirectory: ACLs enabled? true
2023-06-26 13:45:50,699 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2023-06-26 13:45:50,699 INFO namenode.FSDirectory: XAttrs enabled? true
2023-06-26 13:45:50,700 INFO namenode.NameNode: Caching file names occurring more than 10 times
2023-06-26 13:45:50,708 INFO namenode.ReencryptionHandler: Configured throttleLimitHandlerRatio=1.0 for re-encryption
2023-06-26 13:45:50,717 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2023-06-26 13:45:50,720 INFO snapshot.SnapshotManager: SkipList is disabled
2023-06-26 13:45:50,748 INFO util.GSet: Computing capacity for map cachedBlocks
2023-06-26 13:45:50,748 INFO util.GSet: VM type = 64-bit
2023-06-26 13:45:50,748 INFO util.GSet: 0.25% max memory 864 MB = 2.2 MB
2023-06-26 13:45:50,749 INFO util.GSet: capacity = 2^18 = 262144 entries
2023-06-26 13:45:50,764 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2023-06-26 13:45:50,764 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2023-06-26 13:45:50,764 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2023-06-26 13:45:50,768 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2023-06-26 13:45:50,768 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2023-06-26 13:45:50,771 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2023-06-26 13:45:50,771 INFO util.GSet: VM type = 64-bit
2023-06-26 13:45:50,771 INFO util.GSet: 0.029999999329447746% max memory 864 MB = 265.4 KB
2023-06-26 13:45:50,771 INFO util.GSet: capacity = 2^15 = 32768 entries
2023-06-26 13:45:50,774 INFO namenode.FSNamesystem: Removal of Expired Lease on Open Files is enabled
2023-06-26 13:45:50,811 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1979673447-172.31.25.102-1687787150800
2023-06-26 13:45:50,824 INFO common.Storage: Storage directory /mnt/namenode has been successfully formatted.
2023-06-26 13:45:50,826 INFO common.Storage: Storage directory /mnt1/namenode has been successfully formatted.
2023-06-26 13:45:50,868 INFO namenode.FSImageFormatProtobuf: Saving image file /mnt1/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
2023-06-26 13:45:50,868 INFO namenode.FSImageFormatProtobuf: Saving image file /mnt/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
2023-06-26 13:45:51,228 INFO namenode.FSImageFormatProtobuf: Image file /mnt1/namenode/current/fsimage.ckpt_0000000000000000000 of size 403 bytes saved in 0 seconds .
2023-06-26 13:45:51,230 INFO namenode.FSImageFormatProtobuf: Image file /mnt/namenode/current/fsimage.ckpt_0000000000000000000 of size 403 bytes saved in 0 seconds .
2023-06-26 13:45:51,244 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2023-06-26 13:45:51,301 INFO namenode.FSNamesystem: Stopping services started for active state
2023-06-26 13:45:51,301 INFO namenode.FSNamesystem: Stopping services started for standby state
2023-06-26 13:45:51,305 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2023-06-26 13:45:51,306 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2023-06-26 13:45:51,306 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ip-172-31-25-102.eu-central-1.compute.internal/172.31.25.102
************************************************************/
Loading