Enable predicate invention with both geometric and VLM predicates #1666
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This basically just calls both the method to generate predicates from a grammar and the method to generate predicates from a VLM and then combines the atom datasets produced by both.
This does not yet work "well" in BurgerEnv yet because we don't get the predicates we want in the grammar (we'll have to tweak the grammar a bit, and also be careful about what predicates we include/exclude). But the pipeline works in that the atom datasets are combined properly into something with both geometric and VLM ground atoms.
An important change in this is that we now make it possible to replace goal predicates (e.g. goal predicates that we might expect the demonstrator to use) with agent-specific goal predicates -- which is relevant when we want the agent to invent the corresponding goal predicate that the demonstrator used. Before, this was partially done by making it so that tasks' goals could be reframed in terms of alternative goal predicates, but this PR also makes this happen for the predicates that get passed into an approach's initial/known predicates.
Tests
At the time of this writing, (2024-06-14), there is no coverage for
offline_data_method="demo_with_vlm_imgs"
andoffline_data_method="saved_vlm_img_demos_folder"
because a VLM would need to be instantiated and called and that would be slow (there's already a comment in the code about this) -- the current code tests this separately and ignores the coverage issue?