[Discuss] Avoiding duplication of ECS field definitions

[andrewkroh]

Currently each dataset duplicates the definitions of the ECS fields that it uses (data type, descriptions, examples, etc). This puts a burden on the package maintainers to copy the ECS definitions into the dataset and keep it in sync with ECS.

It would be simpler to develop and maintain a package if the dataset only required listing the names of ECS fields that the module uses (and the ECS version). When the package is build the full field definitions for the specified fields can be imported from ECS.

The dataset would declare:

• ECS version
• List of ECS fields (maybe allowing for patterns like host.*, but that's part of the discussion)

The build step would create field declarations in YAML format that conform the package spec.

[mtojek]

/cc @ruflin @ph @ycombinator

Let me transfer this issue to the package-spec repository, which is the right place to talk about the format changes.

[andrewkroh]

I'm not sure this affects the specification. Assuming that the build step used in elastic/integrations continues to generate packages that comply to the package-spec then I assume no changes are necessary.

[mtojek]

It depends on the results of the discussion, we may consider to change the fields file format to contain references (pointers?) to ECS fields instead of duplicate entries.

Regarding the build step, the long term plan is to get rid of the Magefile in integrations repo and 100% depend on https://github.com/elastic/elastic-package . Ideally the integrations will remain a repository with config files without any Go code inside. The builder tools will be totally separated from the source of integrations.

[mtojek]

@ycombinator and I had a discussion about possible mitigation of ECS duplication. One of the ideas we talked about bases on introducing a new JSON/YAML file in <data_stream>/_dev directory with listed ECS fields. This file would be processed during the build step and will result in the generated fields/ecs.yml with generated ECS content using the versioned fields from the ECS repository.

The _dev is an extension to the package specification format that contains resources used to build or test packages, not strictly package assets like fields, configs, images, etc.

[ruflin]

If we add this to the package spec, I think it should not be a requirement. A dev that still wants to create all fields inside the dataset should be able to. I'm ok to make it a generic feature but would at first iterate in integrations on it.

[mtojek]

If we add this to the package spec, I think it should not be a requirement.

It must be in the package-spec to preserve the standard across all packages. It's not strictly the package specification, but the _dev extension. We decided to define _dev together in the package specification and not the create separate on and depend on the package-spec (KISS).

A dev that still wants to create all fields inside the dataset should be able to

Regarding the requirement, I'm rather fan of hard requirements which make the standard stricter. Let's say the ecs.yml is restricted and is only generated during the build step. I'm open for discussion, but this will help us to solve the original problem. Unless developers should modify/overwrite the ECS field definitions.

I'm ok to make it a generic feature but would at first iterate in integrations on it.

The feature will appear only in integrations as the _dev folder is not copied to the package-storage.

[ruflin]

I think we are talking about the same thing? If a developer builds a package outside integrations in repo "foo", elastic-package can be used but no _dev directory must be added, it is optional? This dev just must define all his fields in some *.yml file?

[mtojek]

elastic-package can be used but no _dev directory must be added, it is optional.

Yes, the _dev directory is optional.

This dev just must define all his fields in some *.yml file?

I suppose so, but I'm wondering about the ecs.yml, whether it MUST be always generated (in pair with ECS field names in _dev). I'd like to hear more voices about it.

[webmat]

I agree with the idea of having an automated way to fetch ECS fields to reduce the manual work. By name or by glob would sound like a reasonable first step. Thanks for opening this, @andrewkroh!

We might also want eventually to allow the removal of certain globs, as a second pass. E.g. include source.* but exclude source.as.*. We do have some field sets that have grown pretty big, and I understand that one of the design goals of packages is to have only the fields that are absolutely needed, and no more. 👍

I also agree with @ruflin that using this mechanism should be optional. Some packages may have nothing to do with what ECS covers.

Some packages might also want to override ECS fields. I think this is also acceptable. Legit reasons of doing so: clarifying a description in the context of this package, or adopting experimental ECS changes that aren't yet in the main ECS spec.

How this gets implemented in the package build process is up to you. But it's worth pointing out that the tools used by the ECS team can already help accomplish some of this (see USAGE.md). If this is helpful, we're happy to help you adopt our tool to deliver this, add features you need & so on. If you'd rather implement this from scratch in the package build, let us know if we can help :-)

[ruflin]

Some packages might also want to override ECS fields. I think this is also acceptable. Legit reasons of doing so: clarifying a description in the context of this package

This is spot on and one of the reasons we redefine it every time. It is not really about overwriting the field itself, but the information like title and description to make it specific to the dataset.

[jalvz]

It would be great to extend the idea to avoid duplication of any fields (ECS or non ECS)

[mtojek]

Let me propose a solution (healthy one).

There are few problems around field duplication we need to address:

1. Keep field definitions in sync with ECS (the definition is always the same across all packages).
2. We don't want to install all fields in Elasticsearch - 99% of them are not required.
3. ECS field updates (patch, minor?) shouldn't require rebuilding or redistributing package.
4. We don't want to create a special repository with same/similar content as https://github.com/elastic/ecs

Solution:

The solution consists of following items:

• Auto-generate ECS package out of the http://github.com/elastic/ecs repository
• Add field annotations: lazy (bool), external (bool)
• Add property "dependencies" to the package manifest (NPM style)

Field annotations:

1. Lazy field (bool) - it means that the field definition won't be installed immediately, but it may be installed if other package asks for it.
2. External field (bool) - it means that the field is defined in one of package dependencies.

Manifest properties:

1. Dependencies - a map that contains package dependencies in the NPM format (package name + version constraint)

Comments:

Frankly speaking I would be for delegating responsibility of resolving dependency graph to the Kibana than including it
in the build phase of a package and I understand that it's not a simple task to enable dependency management there.
I have a feeling that we can leverage from it in the future.

Counter-argument to the build phase:

Sure, we can resolve dependencies during the package build time, but it means that all packages in the Package Storage
will end up with duplicated fields (larger repo size) and packages that are contributed directly to the Package Storage won't be able
to use this mechanism.