Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QueryPerformanceRecorder: Group Batched Operations as a Single Query #4760

Merged
merged 31 commits into from
Nov 18, 2023

Conversation

nbauernfeind
Copy link
Member

@nbauernfeind nbauernfeind commented Nov 2, 2023

Fixes #4696.
Fixes #4765.

Change Highlights:

  • Introduce a parent-child relationship for EvaluationNumber on the QueryPerformanceLog enabling a batch query to kick off multiple sub queries.
  • Introduce a parent-child relationship for OperationNumber on the QueryOperationPerfomanceLog enabling easier interaction for users.
  • Convert all EvaluationNumber usages (and blink-table columns) to Long.
  • Convert PerformanceEntry#id to a long (and propagates to blink-table columns).
  • I've removed the evaluation number / operation number from the Performance Log Logger as these are accessible from the QueryPerformanceNugget directly.

Nightlies: https://github.com/nbauernfeind/deephaven-core/actions/runs/6900780591/

Here is a script that can be used to create the hierarchical table view -- perhaps it should be available to all via an EngineMetrics accessor method.

Groovy:

import io.deephaven.engine.table.impl.util.PerformanceQueries
qptt = PerformanceQueries.queryPerformanceAsTreeTable()
qoptt = PerformanceQueries.queryOperationPerformanceAsTreeTable()

Python:

from deephaven import perfmon
qptt = perfmon.query_performance_tree_table()
qoptt = perfmon.query_operation_performance_tree_table()

And for fun here's a lovely screen shot using the java-client TableManagerExample which sends several batches:
Screenshot 2023-11-09 at 1 04 34 PM

@nbauernfeind nbauernfeind added query engine core Core development tasks NoDocumentationNeeded ReleaseNotesNeeded Release notes are needed labels Nov 2, 2023
@nbauernfeind nbauernfeind added this to the October 2023 milestone Nov 2, 2023
@nbauernfeind nbauernfeind self-assigned this Nov 2, 2023
py/server/deephaven/perfmon.py Show resolved Hide resolved
py/server/deephaven/perfmon.py Outdated Show resolved Hide resolved
py/server/deephaven/perfmon.py Outdated Show resolved Hide resolved
py/server/deephaven/perfmon.py Outdated Show resolved Hide resolved
chipkent
chipkent previously approved these changes Nov 15, 2023
Copy link
Member

@chipkent chipkent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python LGTM. Did not review the rest. Let me know if I need to.

Copy link
Member

@rcaudy rcaudy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments on two commits

Copy link
Member

@rcaudy rcaudy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.

@nbauernfeind nbauernfeind merged commit 97b8407 into deephaven:main Nov 18, 2023
12 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Nov 18, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
core Core development tasks DocumentationNeeded query engine ReleaseNotesNeeded Release notes are needed
Projects
None yet
3 participants