Add Latest Transform - Wiz Vulnerabilities #10895

CohenIdo · 2024-08-27T12:20:13Z

solves:

https://github.com/elastic/security-team/issues/9829

Summary

Create a latest Transform for the Wiz vulnerabilities data stream to support CDR.

…sform-wiz

elasticmachine · 2024-08-28T12:10:01Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kcreddy · 2024-08-29T11:27:59Z

@CohenIdo, the CI is failing with error:

error running package asset tests: could not complete test run: can't install the package: there was an apply error: installation failed: can't install the package: could not zip-install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"security_exception\n\tRoot causes:\n\t\tsecurity_exception: action [indices:admin/delete] is unauthorized for service account [elastic/kibana] on indices [security_solution-wiz.vulnerability_latest-v1], this action is granted by the index privileges [delete_index,manage,all]"}

If you added the necessary permissions into Elasticsearch which will be merged into, say 8.15.2 version, then the manifest.yml should be updated with same version so that CI picks up the version you added permissions in:

conditions:
  kibana:
    version: "^8.15.2"

packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/fields/base-fields.yml

packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml

…sform-wiz

maxcold · 2024-09-03T10:03:05Z

packages/wiz/manifest.yml

@@ -1,17 +1,17 @@
 format_version: 3.0.2
 name: wiz
 title: Wiz
-version: "1.7.2"
+version: '1.8.0'


As discussed with @CohenIdo we might want to make this change under preview version first as the UX is not yet ready, plus as the change is quite big and requires more e2e testing

lets put this on preview guys
I don't think we should release this change without a full QA cycle
wdyt?

@kfirpeled yes, me and @maxcold were agreed on that and put it on today sync agenda sync to hear other thoughts. So it seems we all agree. will do

elasticmachine · 2024-09-04T09:17:31Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

.vscode/settings.json

packages/wiz/manifest.yml

efd6 · 2024-09-06T00:12:22Z

packages/wiz/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.8.0-preview01"


Why is this not just a release?

as the change introduces the transform we want to test the approach extensively before opening it up for users. There are also some UX adjustments we might need to do based on the real data coming from Wiz. Making the integration a preview first, will allow the team better DX for testing the features. This approach works well for us in the cloud_security_posture integration

Just to make it clearer, any bug fix can be merged under 1.7.x and any changes that are for 8.16 will be under 1.8.0-preview0x

If bugfixes were continued to be merged under 1.7.x, I think the users will not be seeing them because the mainfest's kibana.version has moved to 8.16.0. The users need to be >= package's kibana.version to get the fixes/upgrades.

efd6 · 2024-09-06T00:13:08Z

packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/fields/fields.yml

+- name: cloud.account.id
+  external: ecs
+- name: cloud.region
+  external: ecs
+- name: package.name
+  external: ecs
+- name: package.version
+  external: ecs
+- name: vulnerability.description
+  external: ecs
+- name: vulnerability.id
+  external: ecs
+- name: vulnerability.score.base
+  external: ecs
+- name: vulnerability.score.version
+  external: ecs
+- name: vulnerability.severity
+  external: ecs


Do these need to be added?

Yes, the transform destination index is not predefined in Elasticsearch (unlike the logs-* and metrics-* indices), so the ECS fields are not inherited automatically and must be explicitly defined.

efd6 · 2024-09-06T00:14:25Z

packages/wiz/manifest.yml

 description: Collect logs from Wiz with Elastic Agent.
 type: integration
 categories:
  - security
  - cloudsecurity_cdr
 conditions:
  kibana:
-    version: "^8.13.0"
+    version: "^8.16.0"


If this is the requirement, this should probably be a draft PR until we get there. If we merge this now, all the subsequent changes will either require 8.16 or have to have back-ports.

the package will be available in Serverless as the stack version in serverless is 8.16 already. Together with the preview version, this should give the team including our product an easier way to test the new approach we are trying out on Wiz, see https://docs.google.com/document/d/1govaQj_8LwUOLtk3zrcjCEvRgyKWRuC-TUihCrk04ao/edit for more details
Are there changes planned for Wiz integration where backporting a concern?

Are there changes planned for Wiz integration where backporting a concern?

It's difficult to predict bugs.

I think unexpected bug fixes can be backported to 1.7, I assume the backporting flow exists for this reason. The bumping kibana stack version while using preview versions works quite well for cloud_security_posture, more so with the introduction of serverless. Any concern with this approach for Wiz except the potential backports?

Can you explain what the feature is present in 8.16 that is not present in 8.13? If there is none, what is the reason for wanting to bump this?

Can you explain what the feature is present in 8.16 that is not present in 8.13? If there is none, what is the reason for wanting to bump this?

This is the change/feature related to 8.16 thats needed: elastic/elasticsearch#112192.
The transform in this PR needs permissions to create and maintain the latest indices. These permissions are targeted for v8.16.0 in Elasticsearch.

thanks @kcreddy , that's correct. Plus we release UX features that would leverage the latest transform results, without them having transform would be just a waste of resources. and yes, won't event work properly due to missing privileges

I'm going to remove my block, but despite the documented backport release workflow, in practice, making this change will in all likelihood result in reduction in fixes being released for pre-8.16-dependent versions of the package. This is just an observation on normal dev responses to additional friction from this kind of thing.

@efd6 thanks for providing more context for your concerns. If I understand correctly, the concerns are not about fixing specific Wiz bugs, but rather about making overall improvements to multiple integrations which is hard then to backport to each of these integrations one by one. I understand this concern, and I think it is a very valid one. But still, I think it's important to have a way not to keep the prs open for months. With serverless and with increased gap between versions it becomes a must in my opinion.

…sform-wiz

efd6

Hold purely to avoid a merge before the version issue is discussed within the team.

CohenIdo · 2024-09-10T09:03:14Z

Hold purely to avoid a merge before the version issue is discussed within the team.

There is a simple way to backport for the pervious stack version.
Adding @maxcold's investigations findings:
"all the docs I could find is this issue elastic/package-spec#225 (linked in the package-spec) To my understanding, our use of the pre-release version is totally valid. I also don't see a problem with backport flow after I did backport for csp integration. It involves a bit more work (create backport branch via CI and remember to update the changelog in main manually) but I don't see it as a big blocker"

kcreddy · 2024-09-11T07:28:37Z

More details on supporting bugfixes on older package versions: https://www.elastic.co/guide/en/integrations-developer/current/developer-workflow-support-old-package.html

packages/wiz/changelog.yml

elasticmachine · 2024-09-11T19:14:09Z

💚 Build Succeeded

Buildkite Build
Commit: b75bcfd

History

💚 Build #15691 succeeded 4a7732a
💚 Build #15637 succeeded 4c3f8f9
💚 Build #15619 succeeded 9f77a15
💚 Build #15518 succeeded 49acef6
💔 Build #15435 failed 28d6aae
💔 Build #15376 failed 2828c70

elastic-sonarqube · 2024-09-11T19:14:13Z

Quality Gate failed

Failed conditions
5.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

relented

kcreddy

LGTM

elasticmachine · 2024-09-12T07:34:37Z

Package wiz - 1.9.0-preview01 containing this change is available at https://epr.elastic.co/search?package=wiz

CohenIdo and others added 6 commits August 27, 2024 10:06

update data stream for CDR

ffedcde

uptate changelog

3392d37

uptate mappings

0b37343

WIP

62c4649

Update test generated files and update readme

138e558

Merge branch 'update-wiz-vul-ds' into add-latest-vul-transform-wiz

cf1325c

CohenIdo mentioned this pull request Aug 27, 2024

[Cloud Security][CDR] Update Wiz vuln privileges elastic/elasticsearch#112192

Merged

andrewkroh added the Integration:wiz Wiz label Aug 27, 2024

CohenIdo added 3 commits August 27, 2024 15:33

update index name

b152e20

Merge remote-tracking branch 'upstream/main' into add-latest-vul-tran…

368e3f3

…sform-wiz

wip

9cca808

andrewkroh added the enhancement New feature or request label Aug 28, 2024

CohenIdo marked this pull request as ready for review August 28, 2024 11:54

CohenIdo requested a review from a team as a code owner August 28, 2024 11:54

update readme

c09d118

andrewkroh added the Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] label Aug 28, 2024

CohenIdo added 5 commits August 28, 2024 15:49

add cloud account name

263dfcc

update tests files

307c49a

update readme

0d55419

update test

a8c8dec

update readme

3c0e0b9

maxcold reviewed Aug 30, 2024

View reviewed changes

packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/fields/base-fields.yml Outdated Show resolved Hide resolved

maxcold reviewed Aug 30, 2024

View reviewed changes

packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml Show resolved Hide resolved

maxcold reviewed Aug 30, 2024

View reviewed changes

packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml Show resolved Hide resolved

elastic deleted a comment from maxcold Sep 2, 2024

CohenIdo commented Sep 2, 2024

View reviewed changes

packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/transform.yml Show resolved Hide resolved

CohenIdo added 2 commits September 2, 2024 14:42

code review comments

f3ef6f8

Merge remote-tracking branch 'upstream/main' into add-latest-vul-tran…

34621ec

…sform-wiz

maxcold reviewed Sep 3, 2024

View reviewed changes

add ecs fields

2828c70

CohenIdo requested a review from kfirpeled September 4, 2024 07:38

update version

28d6aae

CohenIdo requested a review from a team as a code owner September 4, 2024 07:40

maxcold reviewed Sep 5, 2024

View reviewed changes

.vscode/settings.json Outdated Show resolved Hide resolved

maxcold reviewed Sep 5, 2024

View reviewed changes

packages/wiz/manifest.yml Outdated Show resolved Hide resolved

revert undesired changes

49acef6

efd6 reviewed Sep 6, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/main' into add-latest-vul-tran…

9f77a15

…sform-wiz

CohenIdo requested a review from efd6 September 8, 2024 06:25

build fixes

4c3f8f9

efd6 previously requested changes Sep 9, 2024

View reviewed changes

CohenIdo added 3 commits September 10, 2024 09:57

Merge upstream/main, resolving conflicts by taking upstream changes

cdaa24a

update version

4ec1ae7

update stack version

4a7732a

CohenIdo requested a review from efd6 September 10, 2024 09:00

CohenIdo requested a review from kcreddy September 10, 2024 09:08

andrewkroh reviewed Sep 11, 2024

View reviewed changes

packages/wiz/changelog.yml Outdated Show resolved Hide resolved

update changelog

b75bcfd

CohenIdo requested a review from andrewkroh September 11, 2024 18:46

kcreddy approved these changes Sep 12, 2024

View reviewed changes

CohenIdo merged commit b659fac into elastic:main Sep 12, 2024
4 of 5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Latest Transform - Wiz Vulnerabilities #10895

Add Latest Transform - Wiz Vulnerabilities #10895

CohenIdo commented Aug 27, 2024 •

edited

Loading

elasticmachine commented Aug 28, 2024

kcreddy commented Aug 29, 2024 •

edited

Loading

maxcold Sep 3, 2024

kfirpeled Sep 3, 2024

CohenIdo Sep 4, 2024 •

edited

Loading

elasticmachine commented Sep 4, 2024 •

edited

Loading

efd6 Sep 6, 2024

maxcold Sep 6, 2024

CohenIdo Sep 9, 2024

kcreddy Sep 9, 2024

efd6 Sep 6, 2024

CohenIdo Sep 8, 2024

efd6 Sep 6, 2024

maxcold Sep 6, 2024

efd6 Sep 8, 2024

maxcold Sep 9, 2024

efd6 Sep 10, 2024

kcreddy Sep 11, 2024

maxcold Sep 11, 2024

efd6 Sep 11, 2024

maxcold Sep 12, 2024

efd6 left a comment

CohenIdo commented Sep 10, 2024

kcreddy commented Sep 11, 2024

elasticmachine commented Sep 11, 2024

elastic-sonarqube bot commented Sep 11, 2024

kcreddy left a comment

elasticmachine commented Sep 12, 2024

		@@ -1,4 +1,9 @@
		# newer versions go on top
		- version: "1.8.0-preview01"

Add Latest Transform - Wiz Vulnerabilities #10895

Add Latest Transform - Wiz Vulnerabilities #10895

Conversation

CohenIdo commented Aug 27, 2024 • edited Loading

Summary

elasticmachine commented Aug 28, 2024

kcreddy commented Aug 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CohenIdo Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Sep 4, 2024 • edited Loading

🚀 Benchmarks report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efd6 left a comment

Choose a reason for hiding this comment

CohenIdo commented Sep 10, 2024

kcreddy commented Sep 11, 2024

elasticmachine commented Sep 11, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Sep 11, 2024

Quality Gate failed

kcreddy left a comment

Choose a reason for hiding this comment

elasticmachine commented Sep 12, 2024

CohenIdo commented Aug 27, 2024 •

edited

Loading

kcreddy commented Aug 29, 2024 •

edited

Loading

CohenIdo Sep 4, 2024 •

edited

Loading

elasticmachine commented Sep 4, 2024 •

edited

Loading