Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable predicate invention with both geometric and VLM predicates #1666

Merged
merged 11 commits into from
Jun 17, 2024

Conversation

ashayathalye
Copy link
Collaborator

@ashayathalye ashayathalye commented Jun 13, 2024

Summary

This basically just calls both the method to generate predicates from a grammar and the method to generate predicates from a VLM and then combines the atom datasets produced by both.

This does not yet work "well" in BurgerEnv yet because we don't get the predicates we want in the grammar (we'll have to tweak the grammar a bit, and also be careful about what predicates we include/exclude). But the pipeline works in that the atom datasets are combined properly into something with both geometric and VLM ground atoms.

An important change in this is that we now make it possible to replace goal predicates (e.g. goal predicates that we might expect the demonstrator to use) with agent-specific goal predicates -- which is relevant when we want the agent to invent the corresponding goal predicate that the demonstrator used. Before, this was partially done by making it so that tasks' goals could be reframed in terms of alternative goal predicates, but this PR also makes this happen for the predicates that get passed into an approach's initial/known predicates.

Tests

At the time of this writing, (2024-06-14), there is no coverage for offline_data_method="demo_with_vlm_imgs" and offline_data_method="saved_vlm_img_demos_folder" because a VLM would need to be instantiated and called and that would be slow (there's already a comment in the code about this) -- the current code tests this separately and ignores the coverage issue?

@ashayathalye ashayathalye marked this pull request as ready for review June 14, 2024 22:24
@ashayathalye ashayathalye changed the title Geo and vlm preds Enable predicate invention with both geometric and VLM predicates Jun 14, 2024
Copy link
Member

@NishanthJKumar NishanthJKumar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Great work!

@NishanthJKumar NishanthJKumar merged commit 97b9b23 into master Jun 17, 2024
6 checks passed
@NishanthJKumar NishanthJKumar deleted the geo-and-vlm-preds branch June 17, 2024 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants