Merge remote-tracking branch 'upstream/main' into add_fields

.github/CODEOWNERS

@@ -171,7 +171,7 @@
 /packages/digital_guardian @elastic/security-service-integrations
 /packages/docker @elastic/obs-cloudnative-monitoring
 /packages/elastic_agent @elastic/elastic-agent
-/packages/elastic_connectors @elastic/ingestion-team
+/packages/elastic_connectors @elastic/search-extract-and-transform
 /packages/elastic_package_registry @elastic/ecosystem
 /packages/elasticsearch @elastic/stack-monitoring
 /packages/enterprisesearch @elastic/stack-monitoring

packages/abnormal_security/_dev/build/docs/README.md

@@ -8,21 +8,21 @@ The Abnormal Security integration collects data for AI Security Mailbox (formerl
 
 The Abnormal Security integration collects four types of logs:
 
-**[AI Security Mailbox](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/AI%20Security%20Mailbox%20(formerly%20known%20as%20Abuse%20Mailbox))** - Get details of AI Security Mailbox.
+- **[AI Security Mailbox](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/AI%20Security%20Mailbox%20(formerly%20known%20as%20Abuse%20Mailbox))** - Get details of AI Security Mailbox.
 
-**[Audit](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Audit%20Logs)** - Get details of Audit logs for Portal.
+- **[Audit](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Audit%20Logs)** - Get details of Audit logs for Portal.
 
-**[Case](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Cases)** - Get details of Abnormal Cases.
+- **[Case](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Cases)** - Get details of Abnormal Cases.
 
-**[Threat](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Threats)** - Get details of Abnormal Threat Logs.
+- **[Threat](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Threats)** - Get details of Abnormal Threat Logs.
 
 ## Requirements
 
-Elastic Agent must be installed. For more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
+Elastic Agent must be installed. For more details and installation instructions, please refer to the [Elastic Agent Installation Guide](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
 
 ### Installing and managing an Elastic Agent:
 
-You have a few options for installing and managing an Elastic Agent:
+There are several options for installing and managing Elastic Agent:
 
 ### Install a Fleet-managed Elastic Agent (recommended):
 
@@ -36,7 +36,7 @@ With this approach, you install Elastic Agent and manually configure the agent l
 
 You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.
 
-There are some minimum requirements for running Elastic Agent and for more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#_minimum_requirements).
+Please note, there are minimum requirements for running Elastic Agent. For more information, refer to the  [Elastic Agent Minimum Requirements](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#elastic-agent-installation-minimum-requirements).
 
 ## Setup
 
@@ -53,12 +53,12 @@ There are some minimum requirements for running Elastic Agent and for more infor
 
 ### Enabling the integration in Elastic:
 
-1. In Kibana go to Management > Integrations.
-2. In "Search for integrations" search bar, type Abnormal Security.
-3. Click on the "Abnormal Security" integration from the search results.
-4. Click on the "Add Abnormal Security" button to add the integration.
+1. In Kibana navigate to Management > Integrations.
+2. In "Search for integrations" top bar, search for `Abnormal Security`.
+3. Select the "Abnormal Security" integration from the search results.
+4. Select "Add Abnormal Security" to add the integration.
 5. Add all the required integration configuration parameters, including Access Token, Interval, Initial Interval and Page Size to enable data collection.
-6. Click on "Save and continue" to save the integration.
+6. Select "Save and continue" to save the integration.
 
 **Note**: By default, the URL is set to `https://api.abnormalplatform.com`. We have observed that Abnormal Security Base URL changes based on location so find your own base URL.
 

packages/abnormal_security/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.0.0"
+  changes:
+    - description: Release package as GA.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11210
 - version: "0.1.2"
   changes:
     - description: Handles empty threats arrays correctly, ensuring REST calls continue during subsequent intervals.

packages/abnormal_security/docs/README.md

@@ -8,21 +8,21 @@ The Abnormal Security integration collects data for AI Security Mailbox (formerl
 
 The Abnormal Security integration collects four types of logs:
 
-**[AI Security Mailbox](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/AI%20Security%20Mailbox%20(formerly%20known%20as%20Abuse%20Mailbox))** - Get details of AI Security Mailbox.
+- **[AI Security Mailbox](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/AI%20Security%20Mailbox%20(formerly%20known%20as%20Abuse%20Mailbox))** - Get details of AI Security Mailbox.
 
-**[Audit](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Audit%20Logs)** - Get details of Audit logs for Portal.
+- **[Audit](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Audit%20Logs)** - Get details of Audit logs for Portal.
 
-**[Case](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Cases)** - Get details of Abnormal Cases.
+- **[Case](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Cases)** - Get details of Abnormal Cases.
 
-**[Threat](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Threats)** - Get details of Abnormal Threat Logs.
+- **[Threat](https://app.swaggerhub.com/apis-docs/abnormal-security/abx/1.4.3#/Threats)** - Get details of Abnormal Threat Logs.
 
 ## Requirements
 
-Elastic Agent must be installed. For more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
+Elastic Agent must be installed. For more details and installation instructions, please refer to the [Elastic Agent Installation Guide](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
 
 ### Installing and managing an Elastic Agent:
 
-You have a few options for installing and managing an Elastic Agent:
+There are several options for installing and managing Elastic Agent:
 
 ### Install a Fleet-managed Elastic Agent (recommended):
 
@@ -36,7 +36,7 @@ With this approach, you install Elastic Agent and manually configure the agent l
 
 You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.
 
-There are some minimum requirements for running Elastic Agent and for more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#_minimum_requirements).
+Please note, there are minimum requirements for running Elastic Agent. For more information, refer to the  [Elastic Agent Minimum Requirements](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#elastic-agent-installation-minimum-requirements).
 
 ## Setup
 
@@ -53,12 +53,12 @@ There are some minimum requirements for running Elastic Agent and for more infor
 
 ### Enabling the integration in Elastic:
 
-1. In Kibana go to Management > Integrations.
-2. In "Search for integrations" search bar, type Abnormal Security.
-3. Click on the "Abnormal Security" integration from the search results.
-4. Click on the "Add Abnormal Security" button to add the integration.
+1. In Kibana navigate to Management > Integrations.
+2. In "Search for integrations" top bar, search for `Abnormal Security`.
+3. Select the "Abnormal Security" integration from the search results.
+4. Select "Add Abnormal Security" to add the integration.
 5. Add all the required integration configuration parameters, including Access Token, Interval, Initial Interval and Page Size to enable data collection.
-6. Click on "Save and continue" to save the integration.
+6. Select "Save and continue" to save the integration.
 
 **Note**: By default, the URL is set to `https://api.abnormalplatform.com`. We have observed that Abnormal Security Base URL changes based on location so find your own base URL.
 

packages/abnormal_security/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 3.2.1
 name: abnormal_security
 title: Abnormal Security
-version: 0.1.2
+version: 1.0.0
 description: Collect logs from Abnormal Security with Elastic Agent.
 type: integration
 categories:

packages/authentik/_dev/build/docs/README.md

@@ -8,19 +8,17 @@ The authentik integration collects event, group, and user logs using REST API.
 
 The authentik integration collects three types of logs:
 
-| Log Type                                                                                                         |
-|------------------------------------------------------------------------------------------------------------------|
-| **[Event](https://docs.goauthentik.io/developer-docs/api/reference/events-events-list)**                         |
-| **[Group](https://docs.goauthentik.io/developer-docs/api/reference/core-groups-list)**                           |
-| **[User](https://docs.goauthentik.io/developer-docs/api/reference/core-users-list)**                             |
+- **[Event](https://docs.goauthentik.io/developer-docs/api/reference/events-events-list)**                         
+- **[Group](https://docs.goauthentik.io/developer-docs/api/reference/core-groups-list)**                           
+- **[User](https://docs.goauthentik.io/developer-docs/api/reference/core-users-list)**                             
 
 ## Requirements
 
-Elastic Agent must be installed. For more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
+Elastic Agent must be installed. For more details and installation instructions, please refer to the [Elastic Agent Installation Guide](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
 
 ### Installing and managing an Elastic Agent:
 
-You have a few options for installing and managing an Elastic Agent:
+There are several options for installing and managing Elastic Agent:
 
 ### Install a Fleet-managed Elastic Agent (recommended):
 
@@ -34,7 +32,7 @@ With this approach, you install Elastic Agent and manually configure the agent l
 
 You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.
 
-There are some minimum requirements for running Elastic Agent and for more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#_minimum_requirements).
+Please note, there are minimum requirements for running Elastic Agent. For more information, refer to the  [Elastic Agent Minimum Requirements](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#elastic-agent-installation-minimum-requirements).
 
 ## Setup
 
@@ -44,12 +42,12 @@ There are some minimum requirements for running Elastic Agent and for more infor
 
 ### Enabling the integration in Elastic:
 
-1. In Kibana go to Management > Integrations.
-2. In "Search for integrations" search bar, type authentik.
-3. Click on the "authentik" integration from the search results.
-4. Click on the "Add authentik" button to add the integration.
+1. In Kibana navigate to Management > Integrations.
+2. In "Search for integrations" top bar, search for `Authentik`.
+3. Select the "authentik" integration from the search results.
+4. Select "Add authentik" to add the integration.
 5. Add all the required integration configuration parameters, including API Token, Interval and Page Size to enable data collection.
-6. Click on "Save and continue" to save the integration.
+6. Select "Save and continue" to save the integration.
 
 ## Logs reference
 

packages/authentik/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.0.0"
+  changes:
+    - description: Release package as GA.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11210
 - version: "0.1.0"
   changes:
     - description: Initial release.

packages/authentik/docs/README.md

@@ -8,19 +8,17 @@ The authentik integration collects event, group, and user logs using REST API.
 
 The authentik integration collects three types of logs:
 
-| Log Type                                                                                                         |
-|------------------------------------------------------------------------------------------------------------------|
-| **[Event](https://docs.goauthentik.io/developer-docs/api/reference/events-events-list)**                         |
-| **[Group](https://docs.goauthentik.io/developer-docs/api/reference/core-groups-list)**                           |
-| **[User](https://docs.goauthentik.io/developer-docs/api/reference/core-users-list)**                             |
+- **[Event](https://docs.goauthentik.io/developer-docs/api/reference/events-events-list)**                         
+- **[Group](https://docs.goauthentik.io/developer-docs/api/reference/core-groups-list)**                           
+- **[User](https://docs.goauthentik.io/developer-docs/api/reference/core-users-list)**                             
 
 ## Requirements
 
-Elastic Agent must be installed. For more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
+Elastic Agent must be installed. For more details and installation instructions, please refer to the [Elastic Agent Installation Guide](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
 
 ### Installing and managing an Elastic Agent:
 
-You have a few options for installing and managing an Elastic Agent:
+There are several options for installing and managing Elastic Agent:
 
 ### Install a Fleet-managed Elastic Agent (recommended):
 
@@ -34,7 +32,7 @@ With this approach, you install Elastic Agent and manually configure the agent l
 
 You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.
 
-There are some minimum requirements for running Elastic Agent and for more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#_minimum_requirements).
+Please note, there are minimum requirements for running Elastic Agent. For more information, refer to the  [Elastic Agent Minimum Requirements](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#elastic-agent-installation-minimum-requirements).
 
 ## Setup
 
@@ -44,12 +42,12 @@ There are some minimum requirements for running Elastic Agent and for more infor
 
 ### Enabling the integration in Elastic:
 
-1. In Kibana go to Management > Integrations.
-2. In "Search for integrations" search bar, type authentik.
-3. Click on the "authentik" integration from the search results.
-4. Click on the "Add authentik" button to add the integration.
+1. In Kibana navigate to Management > Integrations.
+2. In "Search for integrations" top bar, search for `Authentik`.
+3. Select the "authentik" integration from the search results.
+4. Select "Add authentik" to add the integration.
 5. Add all the required integration configuration parameters, including API Token, Interval and Page Size to enable data collection.
-6. Click on "Save and continue" to save the integration.
+6. Select "Save and continue" to save the integration.
 
 ## Logs reference
 

packages/authentik/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 3.2.1
 name: authentik
 title: authentik
-version: 0.1.0
+version: 1.0.0
 description: Collect logs from authentik with Elastic Agent.
 type: integration
 categories:

packages/aws/changelog.yml

@@ -4,6 +4,11 @@
     - description: Add aws.metrics_names_fingerprint.
       type: enhancement
       link: https://github.com/elastic/integrations/pull/11239
+- version: "2.25.0-preview01"
+  changes:
+    - description: Add related.entity field.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11115
 - version: "2.25.0"
   changes:
     - description: "Allow @custom pipeline access to event.original without setting preserve_original_event."

packages/aws/data_stream/cloudtrail/_dev/test/pipeline/test-add-user-to-group-json.log-expected.json

@@ -51,6 +51,12 @@
                 "name": "admin"
             },
             "related": {
+                "entity": [
+                    "EX_PRINCIPAL_ID",
+                    "arn:aws:iam::123456789012:user/Alice",
+                    "Bob",
+                    "Alice"
+                ],
                 "user": [
                     "Alice",
                     "Bob"

packages/aws/data_stream/cloudtrail/_dev/test/pipeline/test-assume-role-json.log-expected.json

@@ -93,6 +93,18 @@
                     "info"
                 ]
             },
+            "related": {
+                "entity": [
+                    "AROAIN5ATK5U7KEXAMPLE:JohnRole1",
+                    "arn:aws:sts::111111111111:assumed-role/JohnDoe/JohnRole1",
+                    "Role2WithTags",
+                    "JohnDoe",
+                    "arn:aws:iam::111122223333:role/JohnRole2",
+                    "arn:aws:sts::111111111111:assumed-role/test-role/Role2WithTags",
+                    "arn:aws:iam::111111111111:role/JohnRole1",
+                    "arn:aws:iam::111111111111:role/JohnRole2"
+                ]
+            },
             "source": {
                 "address": "81.2.69.144",
                 "geo": {
@@ -228,6 +240,17 @@
                     "info"
                 ]
             },
+            "related": {
+                "entity": [
+                    "AROAIN5ATK5U7KEXAMPLE:JohnRole1",
+                    "arn:aws:sts::111111111111:assumed-role/JohnDoe/JohnRole1",
+                    "Role2WithTags",
+                    "JohnDoe",
+                    "arn:aws:sts::111111111111:assumed-role/test-role/Role2WithTags",
+                    "arn:aws:iam::111111111111:role/JohnRole1",
+                    "arn:aws:iam::111111111111:role/JohnRole2"
+                ]
+            },
             "source": {
                 "address": "81.2.69.144",
                 "geo": {

packages/aws/data_stream/cloudtrail/_dev/test/pipeline/test-change-password-json.log-expected.json

@@ -44,6 +44,11 @@
                 ]
             },
             "related": {
+                "entity": [
+                    "0123456789012",
+                    "Alice",
+                    "arn:aws:iam::0123456789012:user/Alice"
+                ],
                 "user": [
                     "Alice"
                 ]
@@ -110,6 +115,11 @@
                 ]
             },
             "related": {
+                "entity": [
+                    "0123456789012",
+                    "Alice",
+                    "arn:aws:iam::0123456789012:user/Alice"
+                ],
                 "user": [
                     "Alice"
                 ]

packages/aws/data_stream/cloudtrail/_dev/test/pipeline/test-cloudtrail-digest-json.log-expected.json

@@ -140,6 +140,7 @@
                 "path": "AWSLogs/123456789123/CloudTrail-Digest/us-west-2/2020/09/11/123456789123_CloudTrail-Digest_us-west-2_leh-ct-test_us-west-2_20200911T193649Z.json.gz"
             },
             "related": {
+                "entity": [],
                 "hash": [
                     "10e0872f32fa1d299d0cc98e94d4c88a6a2eada9d9fc3ae6d53dfe8d54c7caf807072f1e1eec47efdeecfcc22483887f8fddfc954ae587fba43e7676b5547f432fa8722ba1c5baa6b233bcb528ce7c01e3748aab8f28c16c024de79da820128b4c9e5ce65e98a9c4e631687ecc89c224a11bb3df06ce441ff740e4ac9fbd41159e77f5863550118284121f193e357866fbd0463faffb56e194af196e35a7675c3bbd0a398f43159343c3f59129d6339a281a8fdb3192f3fffea9bd21dbb0a705ebfae1921f2133aab0ad29522aea6df0828c1780d3f3ed6b8270ab3ba24459916b0fbbe82fba6ff9677bafe7306e0f5edcc0f1508cdb4e36f3e3b30e653e9987"
                 ]