Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove event.original removal processors (integrations AWS & Azure) #10888

Merged
merged 13 commits into from
Sep 23, 2024
Merged

Remove event.original removal processors (integrations AWS & Azure) #10888

merged 13 commits into from
Sep 23, 2024

Conversation

Alphayeeeet
Copy link
Contributor

Please label as enhancement

Proposed commit message

Remove event.original removal processors

  • AWS *
  • Azure *

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues

Review #10072 for additional info

@Alphayeeeet Alphayeeeet requested review from a team as code owners August 26, 2024 16:21
@Alphayeeeet Alphayeeeet marked this pull request as draft August 26, 2024 16:21
@Alphayeeeet Alphayeeeet marked this pull request as ready for review August 26, 2024 16:28
@Alphayeeeet
Copy link
Contributor Author

Please run CI tests

@andrewkroh andrewkroh added Integration:azure Azure Logs Integration:azure_functions Azure Functions Integration:azure_network_watcher_vnet Azure Network Watcher VNet Integration:azure_frontdoor Azure Frontdoor Integration:aws_bedrock Amazon Bedrock Integration:azure_network_watcher_nsg Azure Network Watcher NSG Integration:azure_openai Azure OpenAI Integration:aws AWS Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] Team:obs-ds-hosted-services Label for the Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Aug 26, 2024
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@andrewkroh
Copy link
Member

/test

andrewkroh
andrewkroh approved these changes Aug 26, 2024
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Did you confirm that the minimum version is at least 8.11.0 (similar to #10417 (comment))?

@andrewkroh andrewkroh added the enhancement New feature or request label Aug 26, 2024
@elasticmachine
Copy link

elasticmachine commented Aug 26, 2024 •
edited by elastic-vault-github-plugin-prod bot
Loading

🚀 Benchmarks report

Package aws 👍(8) 💚(4) 💔(7)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
inspector 1919.39 1545.6 -373.79 (-19.47%) 💔
route53_resolver_logs 7042.25 4291.85 -2750.4 (-39.06%) 💔
apigateway_logs 9803.92 5649.72 -4154.2 (-42.37%) 💔
cloudfront_logs 2518.89 1964.64 -554.25 (-22%) 💔
cloudwatch_logs 500000 333333.33 -166666.67 (-33.33%) 💔
ec2_logs 47619.05 34482.76 -13136.29 (-27.59%) 💔
elb_logs 6289.31 5291.01 -998.3 (-15.87%) 💔

Package azure 👍(6) 💚(3) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
auditlogs 2380.95 1828.15 -552.8 (-23.22%) 💔
firewall_logs 1612.9 1305.48 -307.42 (-19.06%) 💔

Package azure_frontdoor 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
waf 4950.5 3484.32 -1466.18 (-29.62%) 💔

Package azure_functions 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
functionapplogs 2074.69 1068.38 -1006.31 (-48.5%) 💔

To see the full report comment with /test benchmark fullreport

@Alphayeeeet
Copy link
Contributor Author

LGTM. Did you confirm that the minimum version is at least 8.11.0 (similar to #10417 (comment))?

@andrewkroh I checked minimum Kibana constraints. However I saw that CI is failing on Azure integration due to permission denied on filesystem. Did I miss anything?

@muthu-mps
Copy link
Contributor

I think azure needs an elastic-package test pipeline -g to re-build the files with event.original.

You need to address this comment to fix the build failure.

@Alphayeeeet
Copy link
Contributor Author

Should be done. Can you please run CI again?

@muthu-mps
Copy link
Contributor

/test

@andrewkroh
Copy link
Member

@elastic/obs-ds-hosted-services, can you please have look at the changes. This gives @custom pipelines access to event.original.

kaiyan-sheng
kaiyan-sheng reviewed Aug 30, 2024
View reviewed changes
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove the tag preserve_original_event then since event.original will always be there?

@andrewkroh
Copy link
Member

No, this tag is still used by Fleet's built-in final_pipeline to determine if it should remove event.original.

@andrewkroh
Copy link
Member

/test

@andrewkroh andrewkroh force-pushed the refactor-remove-event-original-removal-temp branch from ac019b3 to 79220dd Compare September 19, 2024 17:51
@andrewkroh andrewkroh enabled auto-merge (squash) September 19, 2024 17:55
@Alphayeeeet
Copy link
Contributor Author

@andrewkroh Could you run CI again?

@efd6
Copy link
Contributor

efd6 commented Sep 23, 2024

/test

@andrewkroh andrewkroh merged commit bfe769d into elastic:main Sep 23, 2024
3 checks passed
@elasticmachine
Copy link

💚 Build Succeeded

History

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
71.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@elastic-vault-github-plugin-prod
Copy link

Package aws - 2.25.0 containing this change is available at https://epr.elastic.co/search?package=aws

@elastic-vault-github-plugin-prod
Copy link

Package aws_bedrock - 0.9.0 containing this change is available at https://epr.elastic.co/search?package=aws_bedrock

@elastic-vault-github-plugin-prod
Copy link

Package azure - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=azure

@elastic-vault-github-plugin-prod
Copy link

Package azure_frontdoor - 1.10.0 containing this change is available at https://epr.elastic.co/search?package=azure_frontdoor

@elastic-vault-github-plugin-prod
Copy link

Package azure_functions - 0.8.0 containing this change is available at https://epr.elastic.co/search?package=azure_functions

@elastic-vault-github-plugin-prod
Copy link

Package azure_network_watcher_nsg - 0.3.0 containing this change is available at https://epr.elastic.co/search?package=azure_network_watcher_nsg

@elastic-vault-github-plugin-prod
Copy link

Package azure_network_watcher_vnet - 0.3.0 containing this change is available at https://epr.elastic.co/search?package=azure_network_watcher_vnet

@elastic-vault-github-plugin-prod
Copy link

Package azure_openai - 1.4.0 containing this change is available at https://epr.elastic.co/search?package=azure_openai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

@muthu-mps muthu-mps muthu-mps approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request Integration:aws_bedrock Amazon Bedrock Integration:aws AWS Integration:azure_frontdoor Azure Frontdoor Integration:azure_functions Azure Functions Integration:azure_network_watcher_nsg Azure Network Watcher NSG Integration:azure_network_watcher_vnet Azure Network Watcher VNet Integration:azure_openai Azure OpenAI Integration:azure Azure Logs Team:obs-ds-hosted-services Label for the Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Alphayeeeet @elasticmachine @andrewkroh @muthu-mps @efd6 @kaiyan-sheng