Skip to content

Commit

Permalink
add swag test set for the leaderboard
Browse files Browse the repository at this point in the history
  • Loading branch information
Rowan Zellers committed Oct 5, 2018
1 parent ccc1260 commit 6b2ca11
Show file tree
Hide file tree
Showing 2 changed files with 20,010 additions and 1 deletion.
5 changes: 4 additions & 1 deletion data/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,12 @@ Each item in the CSV is an example. It's conveniently in two formats.
## full
In `train_full.csv` or `val_full.csv`: we have both the texts of the endings/contexts, as well as the ordinal labels (likely, unlikely), some identification `video-id`,`fold-ind`, the full context `startphrase` (also, split into `sent1`,`sent2`), and the endings. There's the gold ending `gold-ending` and its source `gold-source` which is either generated or comes from the found completion. There's an ordinal label for the gold as well `gold-type`. We also have 3-4 distractors `distractor-N` and an ordinal label for each one `distractor-N-type.` The reason it's 3 or 4 is that sometimes there were more answers filtered out as gibberish by the annotators. When there is a 4th distractor, it's often of lower quality than the others (ranked the most plausible).


## regular (shuffled)

This could be more interesting for modeling, and it's the way the test data is formatted. For each `startphrase` (also, split into `sent1`,`sent2`) we have 4 endings, and a label which says the correct one.
This could be more interesting for modeling, and it's the way the test data is formatted. For each `startphrase` (also, split into `sent1`,`sent2`) we have 4 endings, and a label which says the correct one. You can use `test.csv` for submission on the leaderboard here: [https://leaderboard.dev.allenai.org/swag/submission/create](https://leaderboard.dev.allenai.org/swag/submission/create)



## More info about gold-source
If the source starts with `gold`, it comes from the found data (from an actual video caption). This is the case for all questions in the val and test sets.
Expand Down
Loading

0 comments on commit 6b2ca11

Please sign in to comment.