Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor AIConfig to Sanitize Input for Goal Parameters #3492

Merged
merged 15 commits into from
Apr 30, 2023

Conversation

lc0rp
Copy link
Contributor

@lc0rp lc0rp commented Apr 28, 2023

Background

Refactor AIConfig to Sanitize Input for Goal Parameters

A user reported errors with remove_color_codes and some ai_settings combinations. Investigations showed that some ai_settings formats can cause goals to load as dicts, not strings.

Specifically, if a goal is defined as:

- Fix the files: Download, analyze and fix dirty_data.txt
# It's interpreted as a `dict`

goal = {"Fix the files": "Download, analyze and fix dirty_data.txt"}

# Not a string:

goal = "Fix the files: Download, analyze and fix dirty_data.txt"

Changes

Details:

  • Modified ai_config.py to correctly handle and sanitize user input for AI goals and convert them to formatted strings, to fix an issue where some specially formatted ai_settings.yaml files were causing goals to load as list[dict]
  • test_ai_config.py includes a test for the sanitize_input function in AIConfig class.

Documentation

No documentation change needed

Test Plan

Tests were updated

PR Quality Checklist

  • [ x] My pull request is atomic and focuses on a single change.
  • [ x] I have thoroughly tested my changes with multiple different prompts.
  • [ x] I have considered potential risks and mitigations for my changes.
  • [ x] I have documented my changes clearly and comprehensively.
  • [ x] I have not snuck in any "extra" small tweaks changes

lc0rp and others added 4 commits April 28, 2023 18:27
The `remove_color_codes` function now accepts any type of input that can be cast to a string. Previously, it was only accepting string input and not casting non-string types to string which was causing errors in some cases.

The changes were made to both logs.py and its corresponding test file.
Details:
- Modified `ai_config.py` to correctly handle and sanitize user input for AI goals and convert them to formatted strings, to fix an issue where some specially formatted ai_settings.yaml files were causing goals to load as list[dict]
- `test_ai_config.py` includes a test for the `sanitize_input` function in `AIConfig` class.
- Removed unnecessary tests from `test_logs.py`
@vercel
Copy link

vercel bot commented Apr 28, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
docs ⬜️ Ignored (Inspect) Visit Preview Apr 30, 2023 4:45am

@codecov
Copy link

codecov bot commented Apr 28, 2023

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.35 🎉

Comparison is base (06ae468) 58.78% compared to head (40055a2) 59.14%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3492      +/-   ##
==========================================
+ Coverage   58.78%   59.14%   +0.35%     
==========================================
  Files          68       68              
  Lines        3062     3062              
  Branches      433      502      +69     
==========================================
+ Hits         1800     1811      +11     
+ Misses       1131     1120      -11     
  Partials      131      131              
Impacted Files Coverage Δ
autogpt/config/ai_config.py 79.36% <100.00%> (+17.46%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@github-actions github-actions bot added size/m and removed size/l labels Apr 29, 2023
Copy link
Member

@Pwuts Pwuts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the way we save to the ai_config.yaml need to be fixed as well? Putting some quotes around all the text items?

@lc0rp
Copy link
Contributor Author

lc0rp commented Apr 29, 2023

Does the way we save to the ai_config.yaml need to be fixed as well? Putting some quotes around all the text items?

It is a possibility. I'll take a look.

However, the prompting sub-community is directly editing the ai_settings.yaml file. Many of them are opting for a static ai_settings.yaml and providing the actual instructions in a separate file that GPT-4 can directly parse: 1, 2.

Spitballing: we could move the goals from AI settings to long-term memory. Or we could keep any user-provided initial instructions as free-form prompts, which is the approach that these projects usually take.

@github-actions github-actions bot added size/l and removed size/m labels Apr 29, 2023
@lc0rp
Copy link
Contributor Author

lc0rp commented Apr 29, 2023

Does the way we save to the ai_config.yaml need to be fixed as well? Putting some quotes around all the text items?

Confirmed. AIConfig.save()'s doing the right thing.

@vercel vercel bot temporarily deployed to Preview April 29, 2023 17:23 Inactive
@vercel vercel bot temporarily deployed to Preview April 30, 2023 01:28 Inactive
collijk
collijk previously approved these changes Apr 30, 2023
@collijk
Copy link
Contributor

collijk commented Apr 30, 2023

Did you run black on your repo? Our linter is flagging a failure due to trailing whitespace in your test file

@vercel vercel bot temporarily deployed to Preview April 30, 2023 02:46 Inactive
@lc0rp
Copy link
Contributor Author

lc0rp commented Apr 30, 2023

Found it manually. Black wasn't detecting it.Squashed.

@collijk collijk merged commit 064ac5c into Significant-Gravitas:master Apr 30, 2023
@lc0rp lc0rp deleted the fix_remove_colors_dict_bug branch July 23, 2023 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants