Create different templates for CSPM and CNVM #1125

jeniawhite · 2023-07-10T19:57:16Z

Summary of your changes

In order to have different permissions and policies, we've resorted to creating different templates.

Related Issues

Fixes: Separate CloudFormation templates #1122

Checklist

I have added tests that prove my fix is effective or that my feature works
I have added the necessary README/documentation (if appropriate)

mergify · 2023-07-10T19:57:52Z

This pull request does not have a backport label. Could you fix it @jeniawhite? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit
NOTE: backport-skip has been added to this pull request.

github-actions · 2023-07-10T20:22:41Z

📊 Allure Report - 💚 No failures were reported.

Result	Count
🟥 Failed	0
🟩 Passed	33
⬜ Skipped	1

oren-zohar

We could use another set of eyes here, I focused mainly on style and linting

oren-zohar · 2023-07-10T21:01:22Z

.github/workflows/publish-cloudformation.yml

-          FILENAME="cloudformation-$VERSION-$DATE.yml"
-          echo "Uploading $FILENAME to S3"
-          aws s3 cp deploy/cloudformation/elastic-agent-ec2.yml s3://elastic-cspm-cft/$FILENAME
+          CNVMFILENAME="cloudformation-cnvm-$VERSION-$DATE.yml"


nit:

Suggested change

CNVMFILENAME="cloudformation-cnvm-$VERSION-$DATE.yml"

CNVM_FILENAME="cloudformation-cnvm-$VERSION-$DATE.yml"

oren-zohar · 2023-07-10T21:01:45Z

.github/workflows/publish-cloudformation.yml

+          echo "Uploading $CNVMFILENAME to S3"
+          aws s3 cp deploy/cloudformation/elastic-agent-ec2-cnvm.yml s3://elastic-cnvm-cft/$CNVMFILENAME
+
+          CSPMFILENAME="cloudformation-cspm-$VERSION-$DATE.yml"


nit:

Suggested change

CSPMFILENAME="cloudformation-cspm-$VERSION-$DATE.yml"

CSPM_FILENAME="cloudformation-cspm-$VERSION-$DATE.yml"

oren-zohar · 2023-07-10T21:03:00Z

.pre-commit-config.yaml

@@ -7,7 +7,7 @@ repos:
      - id: check-added-large-files
      - id: check-yaml
        args: [--allow-multiple-documents, --unsafe]
-        exclude: (^tests/deploy/k8s-cloudbeat-tests/templates/.*)$|(^deploy/cloudformation/elastic-agent-ec2.yml)$
+        exclude: (^tests/deploy/k8s-cloudbeat-tests/templates/.*)$|(^deploy/cloudformation/elastic-agent-ec2-*.yml)$


do we still want to ignore them now that we won't use templating?

oren-zohar · 2023-07-10T21:03:17Z

.pre-commit-config.yaml

+            "-rn", # Only display messages
+            "--rcfile=tests/pylintrc", # Link to your config file
+          ]


please revert the auto IDE formatting changes (unless you think they are needed)

oren-zohar · 2023-07-10T21:03:47Z

.pre-commit-config.yaml

-          --check,
-          --diff,
-        ]
+        args: [--line-length=120, --check, --diff]


please revert the auto IDE formatting changes (unless you think they are needed)

oren-zohar · 2023-07-10T21:03:51Z

.pre-commit-config.yaml


  - repo: local
    hooks:
      - id: yq
        name: format with yq
        entry: sh -c 'for file in "$@"; do yq -i "$file"; done'
        language: system
-        args: [ -i ]
+        args: [-i]


please revert the auto IDE formatting changes (unless you think they are needed)

oren-zohar · 2023-07-10T21:03:56Z

.pre-commit-config.yaml

        files: deploy/cloudformation/.*yml

  - repo: https://github.com/awslabs/cfn-python-lint
    rev: v0.77.5
    hooks:
-    -   id: cfn-python-lint
+      - id: cfn-python-lint


please revert the auto IDE formatting changes (unless you think they are needed)

oren-zohar · 2023-07-10T21:04:53Z

deploy/cloudformation/.gitignore

+elastic-agent-ec2-dev-cspm.yml
+elastic-agent-ec2-dev-cnvm.yml


I think you can use dev-* here instead, but don't you think it would be useful to commit these files and share them?

I've kept the pre-existing logic of ignoring the dev generated yml.
I'm not against pushing the dev templates, but they are auto-generated anyways.

oren-zohar · 2023-07-10T21:07:33Z

deploy/cloudformation/elastic-agent-ec2-cnvm.yml

@@ -3,8 +3,8 @@ AWSTemplateFormatVersion: "2010-09-09"
 Description: Creates an EC2 instance with permissions to run elastic-agent
 Parameters:
  LatestAmiId:
-    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
-    Default: '/aws/service/ami-amazon-linux-latest/al2023-ami-minimal-kernel-default-arm64'
+    Type: "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>"


I feel like there's an IDE formatting war between some of the CF devs (see here for example lol) Please let's solve this instead of committing multiple "->' lines

Syncing with @amirbenun it looks like he manually edited the "->' changes (it looks like that he doesn't use any formatter for yml files).
I'm using the default configurations of VSCode.
If you think that I should change the default formatting, please provide me the configuration that you'd like me to use.

rain fmt seems to be the AWS recommendation.

orestisfl · 2023-07-11T08:06:02Z

deploy/cloudformation/README.md

@@ -10,7 +10,7 @@ The EC2 instance has elastic-agent preinstalled in it using the fleet URL and en
 2. You have AWS CLI installed on your laptop and configured to work with our dev account `elastic-security-cloud-security-dev` (in particular, `~/.aws/config` and `~/.aws/credentials` should be set, check https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html for more information)

 *Steps:*
-1. Install the Vulnerability Management integration on a new agent policy, you might have to check the "Display beta integrations" checkbox.
+1. Install Vulnerability Management integration on a new agent policy, you might have to check the "Display beta integrations" checkbox.


"the" is grammatically correct there :)

deploy/cloudformation/README.md

deploy/cloudformation/elastic-agent-ec2-cspm.yml

deploy/cloudformation/gomain.go

amirbenun · 2023-07-11T09:37:23Z

deploy/cloudformation/config.go

 	Dev                   *devConfig `mapstructure:"DEV"`
+	DeploymentType        string     `mapstructure:"DEPLOYMENT_TYPE"`


I think it should be inside the devConfig because all the "regular" parameters being sent as CloudFormation parameters which will cause an unknown parameter.
Another option is to wrap the CloudFormation parameters under some new object.

Dev is also not a part of the template and yet it exists in the same scope.
I do think that the DeploymentType should be in the root because it is not part of the Dev subsection.
It actually decides what template we will apply eventually.
This is only relevant for use-case when we are running via the gomain.go.

amirbenun · 2023-07-11T09:37:32Z

deploy/cloudformation/README.md

@@ -19,7 +19,6 @@ FLEET_URL="<Elastic Agent Fleet URL>"
 ENROLLMENT_TOKEN="<Elastic Agent Enrollment Token>"
 ELASTIC_ARTIFACT_SERVER="https://artifacts.elastic.co/downloads/beats/elastic-agent" # Replace artifact URL with a pre-release version (BC or snapshot)
 ELASTIC_AGENT_VERSION="<Elastic Agent Version>" # e.g: 8.8.0 | 8.8.0-SNAPSHOT
-# INTEGRATION=CloudSecurityPostureManagement # Defaults to VulnerabilityManagement if not specified


Document the new parameter instead

The DeploymentType is not a template parameter and is only being used by the config.go.

amirbenun · 2023-07-11T09:42:58Z

deploy/cloudformation/gomain.go

+		DEV:  "elastic-agent-ec2-dev-cnvm.yml",
+		PROD: "elastic-agent-ec2-cnvm.yml",
+	},
+}


nit: Create a struct

type flavor struct { Name string DevTemplate string ProdTemplate string }

Discussed together and came to conclusion that we would like to keep the map access dynamic for now.
We can re-evaluate in future if needed.

amirbenun · 2023-07-11T09:43:51Z

deploy/cloudformation/gomain.go

@@ -171,3 +188,13 @@ func createStack(stackName string, templatePath string, params map[string]string
 	log.Printf("Created stack %s", *stackOutput.StackId)
 	return nil
 }
+
+func getTemplatePath(deploymentType string, env string) (string, error) {
+	if deploymentType == "" {


where possible, validations should be in config.go under the validation function.

amirbenun · 2023-07-11T14:46:00Z

deploy/cloudformation/config.go

 	Dev                   *devConfig `mapstructure:"DEV"`
+	DeploymentType        string     `mapstructure:"DEPLOYMENT_TYPE"`


The sanity flow needs to create a CNVM stack, you can add an env var here:

cloudbeat/.github/workflows/test-environment.yml

Line 181 in dc411eb

STACK_NAME: "${{ env.CNVM_STACK_NAME}}"

It will do it by default.

func getTemplatePath(deploymentType string, env string) string { if deploymentType == "" { // Default is CNVM deploymentType = DeploymentTypeCNVM

…niawhite/cloudbeat into evgb-CloudFormationSeperation

Discussed in elastic#1125 (comment) https://github.com/aws-cloudformation/rain

Discussed in #1125 (comment) https://github.com/aws-cloudformation/rain

Templates change

4afc3f4

jeniawhite requested review from a team as code owners July 10, 2023 19:57

jeniawhite requested a review from olegsu July 10, 2023 19:57

mergify bot assigned jeniawhite Jul 10, 2023

mergify bot added the backport-skip label Jul 10, 2023

jeniawhite mentioned this pull request Jul 10, 2023

[cloud_security_posture] Seperate cloud formation templates for CSPM and CNVM elastic/integrations#6899

Merged

4 tasks

oren-zohar requested changes Jul 10, 2023

View reviewed changes

oren-zohar requested a review from orestisfl July 11, 2023 08:01

orestisfl requested changes Jul 11, 2023

View reviewed changes

amirbenun reviewed Jul 11, 2023

View reviewed changes

CR changes

4229e8b

jeniawhite requested review from amirbenun, oren-zohar and orestisfl July 12, 2023 11:39

jeniawhite added 4 commits July 12, 2023 14:39

Merge branch 'main' into evgb-CloudFormationSeperation

2722048

Ignore dev templates

127e892

Merge branch 'evgb-CloudFormationSeperation' of https://github.com/je…

2386fc6

…niawhite/cloudbeat into evgb-CloudFormationSeperation

Remove exclude for cf

bf9d8f7

oren-zohar approved these changes Jul 13, 2023

View reviewed changes

jeniawhite added 4 commits July 16, 2023 13:32

Update config path

86761ca

Merge branch 'main' into evgb-CloudFormationSeperation

a331510

Merge branch 'main' into evgb-CloudFormationSeperation

8c75712

Merge branch 'main' into evgb-CloudFormationSeperation

cb422e1

orestisfl approved these changes Jul 17, 2023

View reviewed changes

Merge branch 'main' into evgb-CloudFormationSeperation

113c3b6

jeniawhite enabled auto-merge (squash) July 17, 2023 17:47

oren-zohar disabled auto-merge July 19, 2023 10:35

oren-zohar merged commit d63abaa into elastic:main Jul 19, 2023

orestisfl added a commit to orestisfl/cloudbeat that referenced this pull request Jul 24, 2023

Format CloudFormation with rain

a121d7b

Discussed in elastic#1125 (comment) https://github.com/aws-cloudformation/rain

orestisfl mentioned this pull request Jul 24, 2023

Format CloudFormation with rain #1150

Merged

orestisfl added a commit that referenced this pull request Jul 24, 2023

Format CloudFormation with rain (#1150)

20ce9c9

Discussed in #1125 (comment) https://github.com/aws-cloudformation/rain

	CNVMFILENAME="cloudformation-cnvm-$VERSION-$DATE.yml"
	CNVM_FILENAME="cloudformation-cnvm-$VERSION-$DATE.yml"

	CSPMFILENAME="cloudformation-cspm-$VERSION-$DATE.yml"
	CSPM_FILENAME="cloudformation-cspm-$VERSION-$DATE.yml"

		elastic-agent-ec2-dev-cspm.yml
		elastic-agent-ec2-dev-cnvm.yml

		Dev *devConfig `mapstructure:"DEV"`
		DeploymentType string `mapstructure:"DEPLOYMENT_TYPE"`

Create different templates for CSPM and CNVM #1125

Create different templates for CSPM and CNVM #1125

Uh oh!

Conversation

jeniawhite commented Jul 10, 2023

Summary of your changes

Related Issues

Checklist

Uh oh!

mergify bot commented Jul 10, 2023

Uh oh!

github-actions bot commented Jul 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Allure Report - 💚 No failures were reported.

Uh oh!

oren-zohar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oren-zohar Jul 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeniawhite Jul 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeniawhite Jul 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jul 10, 2023 •

edited

Loading

oren-zohar Jul 10, 2023 •

edited

Loading

jeniawhite Jul 11, 2023 •

edited

Loading

jeniawhite Jul 11, 2023 •

edited

Loading