Initial commit

README.md

@@ -0,0 +1,13 @@
+# GPT-3: Language Models are Few-Shot Learners
+<!-- TODO: fill this in -->
+[arxiv link](https://arxiv.org)
+> Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting.  For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model.  GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans.  We discuss broader societal impacts of this finding and of GPT-3 in general.
+
+## Contents
+- [175b_samples.jsonl](175b_samples.jsonl) - Unconditional, unfiltered 2048 token samples from GPT-3 with p=.85, t=1.&#12288;
+**CONTENT WARNING:** GPT-3 was trained on arbitrary data from the web, so may contain offensive content and language.
+- [data](data) - Synthetic datasets for word scramble and arithmetic tasks described in the paper.
+- [overlap_frequency.md](overlap_frequency.md) - Samples of 13-gram overlaps between our training data and benchmarks, selected by frequency in the training set.
+
+<!-- TODO: -->
+<!-- ## How to cite -->

overlap_frequency.md

@@ -0,0 +1,145 @@
+The following are samples of of 13-grams that occurred in our filtered Common Crawl dataset and how many times they were seen. The samples show that the most frequent 13-grams tend to be junk, while the least frequent appear to be reasonable content.
+
+### Top 13-grams
+```
+159833  all rights reserved this material may not be published broadcast rewritten or redistributed
+61887   1 2 3 4 5 6 7 8 9 10 11 12 13
+47145   2 3 4 5 6 7 8 9 10 11 12 13 14
+45532   3 4 5 6 7 8 9 10 11 12 13 14 15
+42200   a b c d e f g h i j k l m
+36025   e f g h i j k l m n o p q
+35635   b c d e f g h i j k l m n
+33933   4 5 6 7 8 9 10 11 12 13 14 15 16
+33532   c d e f g h i j k l m n o
+33516   d e f g h i j k l m n o p
+32437   associated press all rights reserved this material may not be published broadcast rewritten
+28406   0 0 0 0 0 0 0 0 0 0 0 0 0
+28382   5 6 7 8 9 10 11 12 13 14 15 16 17
+28329   7 8 9 10 11 12 13 14 15 16 17 18 19
+```
+
+### N=1001 samples
+```
+1080    any form or by any means electronic mechanical photocopying recording or otherwise without
+1071    4 4 4 4 4 4 4 4 4 4 4 4 4
+1028    denmark djibouti dominica dominican republic east timor ecuador egypt el salvador equatorial guinea
+1020    name of the father and of the son and of the holy spirit
+1016    shall be deprived of life liberty or property without due process of law
+1014    i m going to go out on a limb here and say that
+1012    creating new collective works for resale or redistribution to servers or lists or
+1008    country can do for you ask what you can do for your country
+1006    1 0 1 0 1 0 1 0 1 0 1 0 1
+1006    your country can do for you ask what you can do for your
+1001    ask not what your country can do for you ask what you can
+```
+
+### N=101 samples
+```
+101     6 chapter 7 chapter 8 chapter 9 chapter 10 chapter 11 chapter 12
+101     place where when you have to go there they have to take you
+101     including the right to reproduce this book or portions thereof in any form
+101     there was with the angel a multitude of the heavenly host praising god
+101     unusual and extraordinary threat to the national security foreign policy and economy of
+101     to this world but be ye transformed by the renewing of your mind
+101     limiting climate change will require substantial and sustained reductions of greenhouse gas emissions
+101     the 16 bit one s complement of the one s complement sum of
+101     observatory is a facility of the national science foundation operated under cooperative agreement
+101     be at the end of the world the angels shall come forth and
+101     you and i want to spend the rest of my life with you
+```
+
+### N=50 samples
+```50      eternal life for god so loved the world that he gave his only
+50      grow not old as we that are left grow old age shall not
+50      higher education in the united states and one of the nine colonial colleges
+50      for the prosecution of persons responsible for serious violations of international humanitarian law
+50      against you in a court of law you have the right to an
+50      2 9 2 10 2 11 2 12 2 13 2 14 2
+50      was given unto them over the fourth part of the earth to kill
+50      those days and also after that when the sons of god came in
+50      the department of the army department of defense or the u s government
+50      heated and transforms into steam within a boiler operating at a high pressure
+50      either die a hero or you live long enough to see yourself become
+50      niger nigeria norway oman pakistan panama papua new guinea paraguay peru philippines poland
+50      not alter our adherence to plos one policies on sharing data and materials
+50      are some of the most studied and detailed periods of human history military
+50      what did you go out into the wilderness to see a reed shaken
+50      consent written informed consent was obtained from the patient for publication of this
+50      electronic or mechanical including photocopying recording or by any information storage or retrieval
+50      is the home rule municipality that is the county seat and the most
+50      intercellular adhesion molecule 1 icam 1 and vascular cell adhesion molecule 1 vcam
+50      5 6 7 8 9 10 11 12 13 14 15 16 the
+50      noble the owner of life savers candy drugstore chain rexall and new york
+50      characters settings etc are the property of their respective owners the original characters
+50      the years in your life that count it s the life in your
+50      work are fictitious any resemblance to real persons living or dead is purely
+50      0 0 0 1 0 0 0 0 1 1 0 0 0
+50      the scientific study of language and involves an analysis of language form language
+50      scientific study of language and involves an analysis of language form language meaning
+50      in dulbecco s modified eagle medium dmem supplemented with 10 fetal bovine serum
+50      they have their exits and their entrances and one man in his time
+50      not matter how slowly you go so long as you do not stop
+50      after that when the sons of god came in unto the daughters of
+50      of wrongs love does not delight in evil but rejoices with the truth
+50      drummer let him step to the music which he hears however measured or
+50      the increase of his government and of peace there will be no end
+50      not delight in evil but rejoices with the truth it always protects always
+50      me i once was lost but now am found was blind but now
+```
+
+### N=21 samples
+```
+21      all worlds begin in darkness and all so end the heart is no
+21      which includes interviewers computer and laboratory technicians clerical workers research scientists volunteers managers
+21      message and we ll get back to you as soon as we can
+21      lady who s sure all that glitters is gold and she s buying
+21      the american academy of arts and sciences and is a fellow of the
+21      has received funding from the european community s seventh framework programme fp7 2007
+21      of international humanitarian law committed in the territory of the former yugoslavia since
+21      and evil there is only power and those too weak to seek it
+21      20 episode 21 episode 22 episode 23 episode 24 episode 25 episode 26
+21      to those who have thrice defied him born as the seventh month dies
+21      grenada montserrat saint kitts and nevis saint lucia saint vincent and the grenadines
+21      b p abbott et al ligo scientific collaboration and virgo collaboration phys rev
+21      although victoria is ranked fourth in terms of gsp per capita because of
+21      you i want to spend the rest of my life with you i
+21      that i happened to be in the right place at the right time
+21      the resulting work only under the same or similar license to this one
+21      cannot be created or destroyed it can only be changed from one form
+21      co operation among states in accordance with the charter of the united nations
+21      this is halloween this is halloween pumpkins scream in the dead of night
+21      gang aft agley an lea e us nought but grief an pain for
+21      i know that i want to spend the rest of my life with
+21      you will also suffer a defeat if you know neither the enemy nor
+21      of thy peace where there is hatred let me sow love where there
+21      israeli war was fought between june 5 and 10 1967 by israel and
+21      secure financial stability facilitate international trade promote high employment and sustainable economic growth
+21      bachelor s and master s degrees in political science from the university of
+21      the which if they should be written every one i suppose that even
+```
+
+### N=1 samples
+```
+1       hunched in its belly till my wet fur froze six miles from earth
+1       fleet and then i m going to save the earth and then just
+1       a and chuang i l 2000 quantum computation and quantum information cambridge university
+1       with all constructed languages gauging the number of speakers of ido is an
+1       idols you made to worship therefore i will send you into exile beyond
+1       to the local bar and began to party as if there was no
+1       jeannette rankin of montana becomes the first female member of the united states
+1       the land lay north of the 36 30 parallel where slavery had been
+1       is explored in articles such as the ultimate the absolute god and zeno
+1       whimper her eyes filled with tears she turned on her heel and ran
+1       perceive if he bends his will thither much of what is passing in
+1       a moment when you say to yourself oh there you are i ve
+1       row 2 in first class stabbed the two unarmed flight attendants who would
+1       for three independent experiments statistical significance was assessed with one way anova and
+1       basic shape the fundamental human end of the universe and here you are
+1       didn t think you d be here i didn t think i d
+1       arora c lund r motwani m sudan and m szegedy proof verification and
+1       of the pines while i sat grilling with my clothes stuck to the
+1       cells were maintained at 37 c in an incubator with a humidified atmosphere
+1       revenge and the allied achaean greek armies nearly lose the war in counterpoint
+1       sat as a chamber articles 26 29 of the statute allow the court
+1       i couldn t save your world i couldn t save any of them
+```