Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Include the scripts for preprocessing OAST and unit tests for chat sft datasets #7112

Merged
merged 23 commits into from
Aug 7, 2023
Merged

Include the scripts for preprocessing OAST and unit tests for chat sft datasets #7112

Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
06c9490
scripts for sft
yidong72 Jul 14, 2023
825a9de
fix style
yidong72 Jul 14, 2023
8d1bc6a
adde special token only for huggingface model
yidong72 Jul 14, 2023
871f068
change default name
yidong72 Jul 14, 2023
b49e619
print out error datapoint content
yidong72 Jul 16, 2023
f3374f7
show error id
yidong72 Jul 16, 2023
199d575
annotation script working
yidong72 Jul 17, 2023
01d49d5
try to be compatible with huggingface tokenizer
yidong72 Jul 17, 2023
726effd
added examples
yidong72 Jul 17, 2023
39192be
added lang
yidong72 Jul 20, 2023
1912e97
added lang
yidong72 Jul 21, 2023
d0c18f0
text to value special case
yidong72 Jul 23, 2023
ca1526b
Merge branch 'main' into sft_scripts_lang
yidong72 Jul 23, 2023
5059a9e
configure the slider
yidong72 Jul 25, 2023
cb67e93
annoatation handles lang
yidong72 Jul 25, 2023
8972863
added the unit test for chat sft dataset
yidong72 Jul 25, 2023
5e9fbc8
used the file in the test dir
yidong72 Jul 26, 2023
f737755
fix json error
yidong72 Jul 27, 2023
d56302d
load local tokenizer
yidong72 Jul 27, 2023
199f900
remove mask count check
yidong72 Jul 27, 2023
cd9331b
Merge branch 'main' into sft_scripts_lang
yidong72 Jul 27, 2023
5016e69
added HF dataset backend
yidong72 Jul 29, 2023
c53662f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'main' into sft_scripts_lang
  • Loading branch information
@yidong72
yidong72 committed Jul 23, 2023
commit ca1526bf5b302b6501f3cdcceee262861b9909ac

This merge commit was added into this branch cleanly.

There are no new changes to show, but you can still view the diff.