Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MCAP cli produces empty message index #862

Closed
mrkbac opened this issue Mar 17, 2023 · 4 comments · Fixed by #873
Closed

MCAP cli produces empty message index #862

mrkbac opened this issue Mar 17, 2023 · 4 comments · Fixed by #873
Labels
bug Something isn't working

Comments

@mrkbac
Copy link

mrkbac commented Mar 17, 2023

Description
The MCAP cli produces an empty message index instead of omitting it. It seems that this slows down Foxglove and the Python lib.

  • Version: CLI v0.0.24
  • Platform: mac OS

Steps To Reproduce
mcap compress on any file

Expected Behavior
no message index at all for topics not in this chunk

@mrkbac mrkbac added the bug Something isn't working label Mar 17, 2023
@foxhubber
Copy link

foxhubber bot commented Mar 17, 2023

Internal tracking ticket: FG-2527

@jtbandes
Copy link
Member

Hi, thanks for the bug report, however, the "steps to reproduce" section is lacking. Can you provide more information about how one can see that this issue has occurred? And can you provide more detail about what slowdown you have observed?

@mrkbac
Copy link
Author

mrkbac commented Mar 18, 2023

Sorry for the unclear explanation, for example the slow down occures in the following code:

with open(sys.argv[1], "rb") as f:
    reader = make_reader(f)
    for schema, channel, message in reader.iter_messages(topics=["/tf_static"]):
        print(f"{channel.topic} ({schema.name}): {message.data}")

even after printing all /tf_static messages the reader keeps reading the whole file.
This only happens when the file was previously processed by the mcap cli (compress, filter, recover, ...), it does not happen for example when the mcap was produced by ros2 bag record
I also noticed slowdowns in foxglove studio when skipping in large files.

I've checked the record Length of the MessageIndex here

if topics is None or summary.channels[channel_id].topic in topics:
out.append(chunk_index)
break

which showed that a MessageIndex for /tf_static exists for all chunks but the record length is almost always zero.

@jtbandes
Copy link
Member

The code snippet is helpful, can you share an example file as well?

james-rms added a commit that referenced this issue Mar 30, 2023
### Public-Facing Changes

Go MCAP writer no longer produces message index records for channels not present in a chunk. This helps indexed readers skip chunks that do not contain any messages on a given channel.

### Description
Fixes #862
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Development

Successfully merging a pull request may close this issue.

2 participants