Skip to content

Fuzzer V2.0.0

Latest
Latest
Compare
Choose a tag to compare
@guy-ps guy-ps released this 01 Aug 11:10
· 3 commits to main since this release
2.0.0
84fcccc
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

Changelog

  • PR - #51
  • Added a custom benchmark interface
  • Added an interface to run only a subset of tests
  • Made some addition to the readme:
  • Documenting the new options
  • Added a link to the google colab
  • Added a function to evaluate response similarity to expected response

Release Notes

  • We added a custom benchmark interface which will allow users to bring their own benchmark to fuzz their system prompt. The benchmark should be in the CSV format and include a “prompt” and “response” columns.
  • We added an interface to run a subset of tests which will help with the speed and efficiency of the system prompt refinement process. Now users can run only a subset of tests iteratively to fix localized problems. It will save both in time and in tokens!
  • We added a response similarity evaluation to several dataset based tests and to the custom benchmark test. This is an upgrade on the previous way we did that, which was to just look for refusal words within the response. This allows for better accuracy when checking the results of testing several prompts and seeing whether they match their expected responses or not.
  • We created a Google Colab notebook! The notebook contains the entire prompt refinement process; from the initial fuzzing through the refinement and localized testing to the regression test and end result. The link is at the top of the README and at the google colab section.
Assets 4
Loading