aws.securityhub_findings: Improve support for CDR

[kcreddy]

Proposed commit message

Improve support for CDR.

• Update securityhub_findings data stream's ingest pipeline to support CDR.
• Update securityhub_findings data stream's mappings according to the new fields.
• Update Findings Summary dashboard with 3 new visualisations using newly added fields.

Fixes: #11040

Note to reviewers: Please DM me for access to the document(s) linked in the issue, it might help in the review.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

cd packages/aws && elastic-package stack down && elastic-package build && elastic-package stack up --version=8.14.0 -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v --data-streams=securityhub_findings

Related issues

• Closes aws.securityhub_findings: Implement mappings for Cloud Security Workflow #11040

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

The issue refers to a document upload, but I cannot find it. So I cannot see whether this follows what has been designed. Is there a reason this is not a public document in the issue?

[efd6]

Is this \n intentional? It won't be expanded since we are in single quotes.

[kcreddy]

Thanks for pointing this out. I've modified it in f898ffc to make it similar to Wiz.

integrations/packages/wiz/data_stream/cloud_configuration_finding/_dev/test/pipeline/test-cloud-configuration-finding.log-expected.json

Line 45 in 4c38387

"remediation": "Follow the step below to ensure that each [Pod](https://kubernetes.io/docs/concepts/workloads/pods) should runs containers with allowed additional capabilities: \r\n* The following capabilities are not allowed : {{removeUnnecessaryCapabilities}} . \r\n* `securityContext.capabilities.drop` key is set to `ALL`. \r\n",

The field is a markdown and should be expanded. Example:

[kcreddy]

@maxcold, does this make sense to use \r\n as delimiter to append remediation.text and remediation.url to derive the field rule.remediation? The outcome is similar to Wiz.

[efd6]

I'm not completely comfortable approving this based on a set of linked proposals that don't have a conclusion (there are items that appear to be still under discussion) and are spread across those documents. IMO a summary acceptance document should be prepared and made public so that users can see what the intended behaviour is, and so that reviewers and maintainers can have truth to base future work on.

[efd6]

Is there is no /, !bang. Is there guaranteed to be one?

[kcreddy]

As per https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html, it is possible that / may not occur. Also, for S3 object, taking the last / would only provide incomplete object prefix.
Hence, I updated to split on : and take the last :. This would mean previous i-0e2ede89308a594d7 would now be instance/i-0e2ede89308a594d7, i.e., an additional <resource-type>/ will be added as prefix. Is this tradeoff okay?
Note that this case of deriving resource.name by / or : will only happen if the .Name is not present in Details.

[CohenIdo]

It seems that some fields (host.name, rule.author) that were mapped here are missing.

[andrewkroh]

IMO a summary acceptance document should be prepared and made public so that users can see what the intended behaviour is, and so that reviewers and maintainers can have truth to base future work on.

++

Logistically, I think a markdown doc in https://github.com/elastic/integrations/tree/main/docs would be ideal for integration developers. It's public, version controlled, and close to the integrations that will implement its guidance. @maxcold, WDYT?

[kcreddy]

It seems that some fields (host.name, rule.author) that were mapped here are missing.

@CohenIdo, The host.name was missed. Actually, the host.name is infact not available when resource is an AwsEc2Instance, but it is present for AwsEcsContainer. I added accordingly.

Also, I missed adding host.ip for AwsEc2Instance which is also added now.
Commit 0897d24 has both changes.

As per clarification from @maxcold, I skipped rule.author

[kcreddy]

IMO a summary acceptance document should be prepared and made public so that users can see what the intended behaviour is, and so that reviewers and maintainers can have truth to base future work on.

++

Logistically, I think a markdown doc in https://github.com/elastic/integrations/tree/main/docs would be ideal for integration developers. It's public, version controlled, and close to the integrations that will implement its guidance.

Regards to #11158 (review):

@maxcold / @CohenIdo, could you make the guide Adapting data stream to Cloud Security in Kibana publicly available in https://github.com/elastic/integrations/tree/main/docs?

[elastic-sonarqube]

Quality Gate failed

Failed conditions
62.9% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: a806320

History

• 💚 Build #16427 succeeded 0897d24
• 💚 Build #16348 succeeded f898ffc
• 💚 Build #16315 succeeded 09a3e99
• 💔 Build #16294 failed 549ea69
• 💚 Build #16287 succeeded a73b971
• 💚 Build #16222 succeeded 866e8c9

cc @kcreddy