[Fleet] Allow snake cased Kibana assets #77515

crob611 · 2020-09-15T16:39:57Z

Summary

This fixes installation of Kibana assets like index-pattern whose asset path in a package will be snake cased index_pattern.

We will now also maintain a kibanaAssetType -> savedObjectType mapping. This is to make sure that any assets in an assets/kibana folder have the appropriate saved object type, since we can't just check against kibanaAssetType anymore.

Finally, a mapping of kibanaAssetType -> installFunction. When we getKibanaAssets now, we get a map of kibanaAssetType -> Array<archive>, so we can act on the kibanaAssetType. Currently, everything is just a saved object, but this is just setting up for the future in case there comes a time when an asset type is not just a simple saved object install, we can introduce new installer methods relatively easily.

I don't think there are any existing packages with index-pattern folders, but if there are we will need to make sure that they get updated at the appropriate time. Should we include a fallback to catch those legacy cases?

elasticmachine · 2020-09-15T16:39:59Z

Pinging @elastic/ingest-management (Team:Ingest Management)

crob611 · 2020-09-15T16:40:14Z

@elasticmachine merge upstream

skh · 2020-09-16T11:26:17Z

Thanks for tackling this!

We generate the index patterns logs-* and metrics-* from the packages' field definitions on the fly whenever a package is installed or removed.

If a package now brings an index pattern with the same name, we run into conflicts. I think we should check for these reserved names, and either silently skip them or return an error.

@neptunian @ruflin what do you think?

ruflin · 2020-09-16T11:58:48Z

There are 2 parts here:

A package is not allowed to contain a Kibana index pattern as part of a dataset
If it contains an index pattern on the global level, ++ on refusing it if it is for a reserved pattern. I would not install the package -> error.

It might even possible that we get this one day in the package-spec to disallow it. @ycombinator

crob611 · 2020-09-16T16:16:42Z

Ok I pushed a new commit that will just not install index-patterns if they are one of the Fleet reserved index patterns.

Is there a way to completely bail in the middle of an install and roll back what has already been done? Throwing doesn't appear to work, which is good because the package will still install if there is an error, so not sure how to bail then.

jfsiii · 2020-09-17T01:13:27Z

@crob611 Thanks for the PR! I'm still reviewing but a 👍 overall. The PR description and code are both clear.

jfsiii · 2020-09-17T19:23:21Z

@elasticmachine merge upstream

jfsiii · 2020-09-17T19:20:44Z

x-pack/plugins/ingest_manager/common/types/models/epm.ts

-export type KibanaAssetReference = Pick<SavedObjectReference, 'id'> & {
-  type: KibanaAssetType;
-};
+export type KibanaAssetReference = Pick<SavedObjectReference, 'id' | 'type'>;


IIUC, SavedObjectReference['type'] is a string vs a KibanaAssetType. I'd like to keep as-is if we can. Was this causing an issue?

@jfsiii KibanaAssetType is now the type that matches file paths, which do not match to saved object types (index_pattern vs index-pattern).

Maybe we bring the KibanaSavedObjectsType over into this file from the kibana/install file and use that type instead?

@crob611 I'll re-read but that seems OK. We definitely want a name and type more refined than string.

@jfsiii OK just pushed a commit that adds a "KibanaSavedObjectType" and uses that type for KibanaAssetReference.

jfsiii · 2020-09-18T10:11:22Z

x-pack/plugins/ingest_manager/server/services/epm/kibana/assets/install.ts

+
+  const resolvedAssets = await Promise.all(assetArrays);
+
+  const result = {} as Record<KibanaAssetType, ArchiveAsset[]>;


Mind moving Record<KibanaAssetType, ArchiveAsset[]> to the function return type?

Can we remove the remove the as Record..., leaving either const result = {} or

Suggested change

const result = {} as Record<KibanaAssetType, ArchiveAsset[]>;

const result: Record<KibanaAssetType, ArchiveAsset[]> = {};

jfsiii · 2020-09-18T10:42:36Z

cc @neptunian for 👀

crob611 · 2020-09-21T17:47:08Z

@elasticmachine merge upstream

crob611 · 2020-09-22T20:49:42Z

@elasticmachine merge upstream

…na-assets

crob611 · 2020-10-08T17:51:12Z

@elasticmachine merge upstream

jfsiii · 2020-11-03T19:47:21Z

@neptunian @skh @ruflin any comments on this? I think we want it and I'd like to merge it soon to avoid conflicts with our pending changes. If you are 👍 on it, I'll merge.

jfsiii · 2020-11-03T20:03:06Z

@elasticmachine merge upstream

Use new `dataTypes` const which replaced `DataType` enum

Remove unused `indexPatternTypes` from outer scope

jfsiii · 2020-11-03T20:46:17Z

@crob611 I merged upstream master and fixed/caused some issues. I think it's good now, but I'll keep an eye on it

fix (?) bad updates from before where new/correct value was used but result wasn't exported

kibanamachine · 2020-11-04T00:30:38Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 5b83a9b

Metrics [docs]

async chunks size

id	before	after	diff
`ingestManager`	1.2MB	1.2MB	-4.0B

page load bundle size

id	before	after	diff
`ingestManager`	386.1KB	386.7KB	+609.0B

History

💔 Build #85667 failed 52daf8b
💔 Build #85648 failed 40cf944
💔 Build #85634 failed 8e774ef
💔 Build #85623 failed c206f21
💚 Build #85205 succeeded 4294a13

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

crob611 · 2020-11-04T19:19:36Z

@jfsiii Looks like this is good to go now. You good with me merging and backporting?

jfsiii · 2020-11-04T19:29:24Z

@crob611 I think so. It passes all tests and has been open for comment for some time.

Let's merge before the changes @skh, @neptunian and I have begun make this even more challenging to merge.

* master: (127 commits) [ILM] Fix breadcrumbs (elastic#82594) [UX]Swap env filter with percentile (elastic#82246) Add platform's missing READMEs (elastic#82268) [Discover] Adding uiMetric to track Visualize link click (elastic#82344) [Search] Add used index pattern name to the search agg error field (elastic#82604) improve client-side SO client get pooling (elastic#82603) [Security Solution] Unskips Overview tests (elastic#82459) Embeddables/migrations (elastic#82296) [Enterprise Search] Refactor product server route registrations to their own files/folders (elastic#82663) Moving reinstall function outside of promise.all (elastic#82672) Load choropleth layer correctly (elastic#82628) Master backport elastic#81233 (elastic#82642) [Fleet] Allow snake cased Kibana assets (elastic#77515) Reduce saved objects authorization checks (elastic#82204) [data.search] Add request handler context and asScoped pattern (elastic#80775) [ML] Fixes formatting of fields in index data visualizer (elastic#82593) Usage collector readme (elastic#82548) [Lens] Visualization validation and better error messages (elastic#81439) [ML] Add annotation markers to time series brush area to indicate annotations exist outside of selected range (elastic#81490) chore(NA): install microdnf in UBI docker build only (elastic#82611) ...

This reverts commit 1cd477a.

…" (elastic#82706)" This reverts commit bc05e79.

* master: Revert "[Fleet] Allow snake cased Kibana assets (elastic#77515)" (elastic#82706)

* Revert "Revert "[Fleet] Allow snake cased Kibana assets (#77515)" (#82706)" This reverts commit bc05e79. * Rename test index pattern

kibanamachine · 2020-11-06T19:54:38Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 77515 or prevent reminders by adding the backport:skip label.

* Revert "Revert "[Fleet] Allow snake cased Kibana assets (elastic#77515)" (elastic#82706)" This reverts commit bc05e79. * Rename test index pattern

* Revert "Revert "[Fleet] Allow snake cased Kibana assets (#77515)" (#82706)" This reverts commit bc05e79. * Rename test index pattern Co-authored-by: Kibana Machine <[email protected]>

kibanamachine · 2020-11-09T20:04:40Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 77515 or prevent reminders by adding the backport:skip label.

kibanamachine · 2020-11-10T20:14:39Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 77515 or prevent reminders by adding the backport:skip label.

Corey Robertson added 3 commits September 11, 2020 12:53

Properly handle kibana assets with underscores in their path

a2469da

Recomment test

a3d768b

Fix type check

06094fa

crob611 added v7.10.0 Team:Fleet Team label for Observability Data Collection Fleet team labels Sep 15, 2020

crob611 requested a review from a team September 15, 2020 16:39

Merge branch 'master' into integrations-kibana-assets

6c9ed86

crob611 added the release_note:skip Skip the PR/issue when compiling release notes label Sep 15, 2020

Don't install index patterns that are reserved

639f90d

jen-huang added the v8.0.0 label Sep 16, 2020

Merge branch 'master' into integrations-kibana-assets

f532be5

jfsiii reviewed Sep 18, 2020

View reviewed changes

Introduce SavedObjectType to use on AssetReference

a9fb9da

elasticmachine and others added 2 commits September 21, 2020 13:47

Merge branch 'master' into integrations-kibana-assets

b05aa37

Fix Test

ae1e0fd

elasticmachine and others added 3 commits September 22, 2020 16:49

Merge branch 'master' into integrations-kibana-assets

0f1e622

Merge branch master

f100adf

Merge remote-tracking branch 'upstream/master' into integrations-kiba…

0ecb8c6

…na-assets

Merge branch 'master' into integrations-kibana-assets

a3919d1

Corey Robertson and others added 2 commits November 2, 2020 09:58

Merge upstream/master

4294a13

Merge branch 'master' into integrations-kibana-assets

c206f21

jfsiii approved these changes Nov 3, 2020

View reviewed changes

kibanamachine and others added 3 commits November 3, 2020 15:03

Merge branch 'master' into integrations-kibana-assets

8e774ef

Update install.ts

f0ecbc4

Use new `dataTypes` const which replaced `DataType` enum

Update install.ts

40cf944

Remove unused `indexPatternTypes` from outer scope

John Schulz added 3 commits November 3, 2020 16:28

Update install.ts

f84206a

fix (?) bad updates from before where new/correct value was used but result wasn't exported

Update install.ts

52daf8b

Update install.ts

5b83a9b

crob611 merged commit 1cd477a into elastic:master Nov 4, 2020

crob611 pushed a commit to crob611/kibana that referenced this pull request Nov 5, 2020

Revert "[Fleet] Allow snake cased Kibana assets (elastic#77515)"

488f074

This reverts commit 1cd477a.

crob611 pushed a commit that referenced this pull request Nov 5, 2020

Revert "[Fleet] Allow snake cased Kibana assets (#77515)" (#82706)

bc05e79

This reverts commit 1cd477a.

crob611 pushed a commit to crob611/kibana that referenced this pull request Nov 5, 2020

Revert "Revert "[Fleet] Allow snake cased Kibana assets (elastic#77515)…

e695e69

…" (elastic#82706)" This reverts commit bc05e79.

gmmorris added a commit to gmmorris/kibana that referenced this pull request Nov 5, 2020

Merge branch 'master' into alerts/assign-action-groups

401d99f

* master: Revert "[Fleet] Allow snake cased Kibana assets (elastic#77515)" (elastic#82706)

jfsiii pushed a commit that referenced this pull request Nov 5, 2020

[Fleet] Remove asterix from test file name (#82721)

6437139

* Revert "Revert "[Fleet] Allow snake cased Kibana assets (#77515)" (#82706)" This reverts commit bc05e79. * Rename test index pattern

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 6, 2020

crob611 added backport:skip This commit does not require backporting and removed backport missing Added to PRs automatically when the are determined to be missing a backport. labels Nov 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Allow snake cased Kibana assets #77515

[Fleet] Allow snake cased Kibana assets #77515

crob611 commented Sep 15, 2020

elasticmachine commented Sep 15, 2020

crob611 commented Sep 15, 2020

skh commented Sep 16, 2020

ruflin commented Sep 16, 2020

crob611 commented Sep 16, 2020

jfsiii commented Sep 17, 2020

jfsiii commented Sep 17, 2020

jfsiii Sep 17, 2020

crob611 Sep 18, 2020

jfsiii Sep 21, 2020

crob611 Sep 21, 2020

jfsiii Sep 18, 2020

jfsiii Nov 3, 2020

jfsiii commented Sep 18, 2020

crob611 commented Sep 21, 2020

crob611 commented Sep 22, 2020

crob611 commented Oct 8, 2020

jfsiii commented Nov 3, 2020 •

edited

Loading

jfsiii commented Nov 3, 2020

jfsiii commented Nov 3, 2020

kibanamachine commented Nov 4, 2020

crob611 commented Nov 4, 2020

jfsiii commented Nov 4, 2020 •

edited

Loading

kibanamachine commented Nov 6, 2020

kibanamachine commented Nov 9, 2020

kibanamachine commented Nov 10, 2020


		const resolvedAssets = await Promise.all(assetArrays);

		const result = {} as Record<KibanaAssetType, ArchiveAsset[]>;

	const result = {} as Record<KibanaAssetType, ArchiveAsset[]>;
	const result: Record<KibanaAssetType, ArchiveAsset[]> = {};

[Fleet] Allow snake cased Kibana assets #77515

[Fleet] Allow snake cased Kibana assets #77515

Conversation

crob611 commented Sep 15, 2020

Summary

elasticmachine commented Sep 15, 2020

crob611 commented Sep 15, 2020

skh commented Sep 16, 2020

ruflin commented Sep 16, 2020

crob611 commented Sep 16, 2020

jfsiii commented Sep 17, 2020

jfsiii commented Sep 17, 2020

jfsiii Sep 17, 2020

Choose a reason for hiding this comment

crob611 Sep 18, 2020

Choose a reason for hiding this comment

jfsiii Sep 21, 2020

Choose a reason for hiding this comment

crob611 Sep 21, 2020

Choose a reason for hiding this comment

jfsiii Sep 18, 2020

Choose a reason for hiding this comment

jfsiii Nov 3, 2020

Choose a reason for hiding this comment

jfsiii commented Sep 18, 2020

crob611 commented Sep 21, 2020

crob611 commented Sep 22, 2020

crob611 commented Oct 8, 2020

jfsiii commented Nov 3, 2020 • edited Loading

jfsiii commented Nov 3, 2020

jfsiii commented Nov 3, 2020

kibanamachine commented Nov 4, 2020

💚 Build Succeeded

Metrics [docs]

async chunks size

page load bundle size

History

crob611 commented Nov 4, 2020

jfsiii commented Nov 4, 2020 • edited Loading

kibanamachine commented Nov 6, 2020

kibanamachine commented Nov 9, 2020

kibanamachine commented Nov 10, 2020

jfsiii commented Nov 3, 2020 •

edited

Loading

jfsiii commented Nov 4, 2020 •

edited

Loading