Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MINOR: Fix docs for end-to-end record latency metrics #10449

Merged
merged 1 commit into from
Mar 31, 2021
Merged

MINOR: Fix docs for end-to-end record latency metrics #10449

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 25 additions & 22 deletions docs/ops.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2453,8 +2453,8 @@ <h5 class="anchor-heading"><a id="kafka_streams_thread_monitoring" class="anchor
</table>

<h5 class="anchor-heading"><a id="kafka_streams_task_monitoring" class="anchor-link"></a><a href="#kafka_streams_task_monitoring">Task Metrics</a></h5>
All of the following metrics have a recording level of <code>debug</code>, except for the dropped-records-*,
active-process-ratio, and record-e2e-latency-* metrics which have a recording level of <code>info</code>:
All of the following metrics have a recording level of <code>debug</code>, except for the dropped-records-* and
active-process-ratio metrics which have a recording level of <code>info</code>:
<table class="data-table">
<tbody>
<tr>
Expand Down Expand Up @@ -2534,31 +2534,19 @@ <h5 class="anchor-heading"><a id="kafka_streams_task_monitoring" class="anchor-l
</tr>
<tr>
<td>active-process-ratio</td>
<td>The total number of records dropped within this task.</td>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes a copy and paste error coming from the dropped-records-* metrics.

<td>kafka.streams:type=stream-task-metrics,thread-id=([-.\w]+),task-id=([-.\w]+)</td>
</tr>
<tr>
<td>record-e2e-latency-avg</td>
<td>The average end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>kafka.streams:type=stream-task-metrics,thread-id=([-.\w]+),task-id=([-.\w]+)</td>
</tr>
<tr>
<td>record-e2e-latency-max</td>
<td>The maximum end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>kafka.streams:type=stream-task-metrics,thread-id=([-.\w]+),task-id=([-.\w]+)</td>
</tr>
<tr>
<td>record-e2e-latency-min</td>
<td>The minimum end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>The fraction of time the stream thread spent on processing this task among all assigned active tasks.</td>
<td>kafka.streams:type=stream-task-metrics,thread-id=([-.\w]+),task-id=([-.\w]+)</td>
</tr>
</tbody>
</table>

<h5 class="anchor-heading"><a id="kafka_streams_node_monitoring" class="anchor-link"></a><a href="#kafka_streams_node_monitoring">Processor Node Metrics</a></h5>
The following metrics are only available on certain types of nodes, i.e., process-rate and process-total are
only available for source processor nodes and suppression-emit-rate and suppression-emit-total are only available
for suppression operation nodes. All of the metrics have a recording level of <code>debug</code>:
The following metrics are only available on certain types of nodes, i.e., the process-* metrics are only available for
source processor nodes, the suppression-emit-* metrics are only available for suppression operation nodes, and the
record-e2e-latency-* metrics are only available for source processor nodes and terminal nodes (nodes without successor
nodes).
All of the metrics have a recording level of <code>debug</code>, except for the record-e2e-latency-* metrics which have
a recording level of <code>info</code>:
<table class="data-table">
<tbody>
<tr>
Expand Down Expand Up @@ -2586,7 +2574,22 @@ <h5 class="anchor-heading"><a id="kafka_streams_node_monitoring" class="anchor-l
<td>The total number of records that have been emitted downstream from suppression operation nodes.</td>
<td>kafka.streams:type=stream-processor-node-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),processor-node-id=([-.\w]+)</td>
</tr>
</tbody>
<tr>
<td>record-e2e-latency-avg</td>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, the record e2e latency metrics are processor node level metrics that are only available on certain types of nodes. Since the bean regex depends on the level (i.e., the metric group) the regex was wrong before.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, thanks. Tbh I had forgotten how we ultimately ended up doing these metrics, thanks for catching this

<td>The average end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>kafka.streams:type=stream-processor-node-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),processor-node-id=([-.\w]+)</td>
</tr>
<tr>
<td>record-e2e-latency-max</td>
<td>The maximum end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>kafka.streams:type=stream-processor-node-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),processor-node-id=([-.\w]+)</td>
</tr>
<tr>
<td>record-e2e-latency-min</td>
<td>The minimum end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>kafka.streams:type=stream-processor-node-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),processor-node-id=([-.\w]+)</td>
</tr>
</tbody>
</table>

<h5 class="anchor-heading"><a id="kafka_streams_store_monitoring" class="anchor-link"></a><a href="#kafka_streams_store_monitoring">State Store Metrics</a></h5>
Expand Down