[CI] Review output in CI builds #2639

mrodm · 2025-06-05T16:32:08Z

Make easier to read Buidkite output:

Add more groups in Buildkite output
Avoid executing some commands in clean up process if the Elastic stack failed to start.
Ensure that the failed group in the Buildkite output is opened (link)

Author's checklist

Validate that parallel and false positive targets run each package in a different Buildkite step
Validate that other targets test all the corresponding packages in the same Buildkite step.
Validate that Serverless pipeline works testing all the packages in the same Buildkite step. https://buildkite.com/elastic/elastic-package-test-serverless/builds/490

mrodm · 2025-06-06T10:29:18Z

/test

mrodm · 2025-06-06T11:21:37Z

/test

mrodm · 2025-07-22T16:46:08Z

test serverless

elastic-vault-github-plugin-prod · 2025-07-22T16:50:44Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/473

mrodm · 2025-07-22T17:17:54Z

test serverless

elastic-vault-github-plugin-prod · 2025-07-22T17:20:39Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/474

mrodm · 2025-07-23T10:04:39Z

test serverless

elastic-vault-github-plugin-prod · 2025-07-23T10:07:32Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/476

mrodm · 2025-07-23T10:56:15Z

test serverless

elastic-vault-github-plugin-prod · 2025-07-23T10:59:19Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/477

mrodm · 2025-07-23T11:16:23Z

test serverless

elastic-vault-github-plugin-prod · 2025-07-23T17:09:20Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/481

mrodm · 2025-07-23T20:00:07Z

/test

mrodm · 2025-07-23T20:40:33Z

.buildkite/scripts/integration_tests.sh

+if [[ "${TARGET}" == "${TEST_BUILD_INSTALL_ZIP_TARGET}" || "${TARGET}" == "${FALSE_POSITIVES_TARGET}" ]]; then
+    echo "--- Install yq"
+    with_yq
+fi


No need to install yq in other targets for now.

mrodm · 2025-07-23T20:54:19Z

scripts/test-build-install-zip-file.sh

@@ -1,17 +1,22 @@
 #!/bin/bash

-set -euxo pipefail
+set -euo pipefail


Removed set -x from scripts/test-build*.sh shell scripts.
It looks to me that it is not needed at least in these targets.

I'm not totally sure if this flag should be kept in the other scripts (e.g. scripts/test-check-packages.sh).

WDYT ? Should we remove set -x in all these scripts ?

Umm, I find these useful when debugging, there would be some way to optionally add them?

The issue in these scripts is that there are some commands that are inside nested loops or nested ifs. That causes that set -x adds three dashes to show the command executed and Buildkite interprets that as a Collapsed group.

I've applied a workaround where it was affected via running the commands in a sub-shell and disabling that flag set +x. After that command, it continues with the previous value (set -x`).

mrodm · 2025-07-23T21:01:31Z

test serverless

elastic-vault-github-plugin-prod · 2025-07-23T21:04:20Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/482

mrodm · 2025-07-23T21:17:27Z

scripts/stack_helpers.sh

+is_stack_created() {
+    local files=0
+    files=$(find ~/.elastic-package -type f -name "docker-compose.yml" | wc -l)
+    if [ "${files}" -gt 0 ]; then
+        return 0
+    fi
+    return 1
+}
+


Helper function to know if elastic-package stack up command has been performed or not.

There could be packages that fail in elastic-package check command and if so, the Elastic stack has not been created.

This is helpful to avoid running commands like elastic-package stack <dump|down> that are going to fail.

mrodm · 2025-07-24T08:44:58Z

scripts/test-build-install-zip.sh

+  # Required containers could not be running, so ignore the error
+  elastic-package stack dump -v --output build/elastic-stack-dump/build-zip || true


Example of the error to try to avoid:
https://buildkite.com/elastic/elastic-package/builds/5786#01983b8a-52f4-4906-a5fe-7379469778ed/350-608

This will help to continue running the other commands in the cleanup function.

elasticmachine · 2025-07-24T09:16:49Z

💔 Build Failed

Buildkite Build
Commit: 80b23a6

Failed CI Steps

:go: Integration test: stack-command-with-self-monitor

History

💔 Build #5789 failed 9b035ed
💔 Build #5787 failed 91303e9
💔 Build #5769 failed 85a83ca
💚 Build #5767 succeeded e26d478
💚 Build #5762 succeeded 2a241b5

cc @mrodm

mrodm · 2025-07-24T10:17:57Z

test serverless

elastic-vault-github-plugin-prod · 2025-07-24T10:20:53Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/490

mrodm · 2025-07-24T13:28:03Z

The error/issue of self-monitor step could be related to this elastic/integrations#14674

mrodm · 2025-07-24T16:20:24Z

The error/issue of self-monitor step could be related to this elastic/integrations#14674

Tested with elastic-package v0.113.0 and it also happens.

Tested by changing the version of the system package to be installed to 2.4.0 and it worked successfully:

--- internal/stack/_static/kibana.yml.tmpl
+++ internal/stack/_static/kibana.yml.tmpl
@@ -56,7 +56,7 @@ xpack.cloudSecurityPosture.enabled: true
 xpack.fleet.packages:
   {{ if eq $self_monitor_enabled "true" }}
   - name: system
-    version: latest
+    version: 2.4.0
   {{ end }}
   - name: elastic_agent
     version: latest

So, I think this failure in the CI is unrelated to the changes in this PR.

mrodm self-assigned this Jun 5, 2025

mrodm added 9 commits July 22, 2025 13:05

Add more collapsed sections in CI output

38e398b

Update collapsed section titles

3c71322

Avoid running stack dump and down if stack is not started

5f91789

Test failure

c774108

Ensure failed group is opened - Buildkite output

18e885f

Run stack down command in any case to shutdown all containers

2c8c36f

Run dump and ensure it does not exit build to continue cleanup process

269a449

Rephrase group title

75831b4

Remove set x flag in build zip scripts

a5ca34a

mrodm force-pushed the update_scripts_output branch from 202d0c3 to a5ca34a Compare July 22, 2025 11:20

mrodm added 2 commits July 22, 2025 17:40

Change to uppercase

7bf29af

yq is not required in Serverless builds

df812de

Add comment

93e4ff1

Install elastic-package just once in Serverless

a314998

mrodm changed the title ~~Add more collapsed sections in CI output~~ [CI] Review output in CI builds Jul 23, 2025

Show just one message to upload safe logs

569ca1c

Fix lint error

5ba8b0f

Ensure all steps in pre-exit are executed

dce92ea

mrodm commented Jul 23, 2025

View reviewed changes

Revert refactor upload safe logs

9a048e1

mrodm force-pushed the update_scripts_output branch from 61a3f54 to 9a048e1 Compare July 23, 2025 21:00

mrodm commented Jul 23, 2025

View reviewed changes

mrodm added 4 commits July 24, 2025 09:41

Test with set -x flag

192ba69

Test adding a subshell for testtype func

91303e9

Use subshell and add more collapsed groups in build-install scripts

7174187

Ignore errors dumping logs in cleanup

9b035ed

mrodm commented Jul 24, 2025

View reviewed changes

mrodm added 4 commits July 24, 2025 10:53

Add set +x in a subshell to get stack arguments

de15917

Add collapsed groups

c990503

Move set flags

6e33bd7

Apply missing set +x to get test type

80b23a6

mrodm marked this pull request as ready for review July 24, 2025 11:32

mrodm requested a review from a team as a code owner July 24, 2025 11:32

mrodm requested a review from jsoriano July 24, 2025 11:51

jsoriano approved these changes Jul 24, 2025

View reviewed changes

mrodm merged commit 3bac9e1 into elastic:main Jul 24, 2025
2 of 3 checks passed

mrodm deleted the update_scripts_output branch July 24, 2025 16:24

mrodm mentioned this pull request Jul 25, 2025

Re-assess output shown in elastic-package when running tests in CI #1903

Closed

		# Required containers could not be running, so ignore the error
		elastic-package stack dump -v --output build/elastic-stack-dump/build-zip \|\| true

[CI] Review output in CI builds #2639

[CI] Review output in CI builds #2639

Uh oh!

Conversation

mrodm commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Author's checklist

Uh oh!

mrodm commented Jun 6, 2025

Uh oh!

mrodm commented Jun 6, 2025

Uh oh!

mrodm commented Jul 22, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 22, 2025

Uh oh!

mrodm commented Jul 22, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 22, 2025

Uh oh!

mrodm commented Jul 23, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 23, 2025

Uh oh!

mrodm commented Jul 23, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 23, 2025

Uh oh!

mrodm commented Jul 23, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 23, 2025

Uh oh!

mrodm commented Jul 23, 2025

Uh oh!

mrodm Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

jsoriano Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm commented Jul 23, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 23, 2025

Uh oh!

mrodm Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

mrodm commented Jul 24, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 24, 2025

Uh oh!

mrodm commented Jul 24, 2025

Uh oh!

mrodm commented Jul 24, 2025

Uh oh!

Uh oh!

Uh oh!

mrodm commented Jun 5, 2025 •

edited

Loading

elasticmachine commented Jul 24, 2025 •

edited

Loading