Learning NLP and Large Language Model (LLM)

• Personal learning path for NLP and Large Language Model (LLM)
• Paper recommendations

Chinese version: README_CN.md (in progress)

NLP Learning Materials

**Notes: **

1. I don't recommend older materials (even if they are "classic") because everything changes fast in NLP
2. You need to know basic knowledge of machine learning and Python
   1. Machine learning courses:
      1. Home | CS 189/289A (eecs189.org)
      2. CS229: Machine Learning (stanford.edu) (theoretic perspective)
   2. Python: too many courses... (ex: CS50's Introduction to Programming with Python (harvard.edu) )

Courses / Tutorials:

• Stanford CS 224N | Natural Language Processing with Deep Learning
• CS224U: Natural Language Understanding - Spring 2023 (stanford.edu)
• Lena Voita (lena-voita.github.io)
• Introduction - Hugging Face NLP Course (good intro about huggingface libraries)

Books:

• Speech and Language Processing (stanford.edu)
• Natural Language Processing with Transformers Book

Articles:

• Intro to Transformer model:

  • Attention? Attention!
  • The Illustrated Transformer
  • Annotated Transformer

• Reading lists:

  • Understanding Large Language Models -- A Transformative Reading List
  • Foundational must read GPT/LLM papers

Important Papers

Note: I haven't read all of the papers in detail so I may change this list frequently

Surveys

• A Survey of Large Language Models
• Challenges and Applications of Large Language Models
• A Survey on Multimodal Large Language Models

Language Model Papers and Reports

• Transformer original paper

• BERT

• DistilBERT

• GPT 1.0

• GPT 2.0

• GPT 3.0

• T5

• FLAN

• InstructGPT

• Flan-T5/PaLM

• GLM-130B

• LLaMA

• LLaMA 2

• GPT-4

• Stanford Alpaca

• Vicuna

• ChatGLM2-6B

• WizardLM

  • WizardCoder

  • WizardMath

• Falcon-7B

• Baichuan2-7B

• Mistral 7B

• Claude 3

LLM Learning and Tuning Techniques

Prompt Tuning

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

In-Context Learning

• A Survey for In-context Learning
• Language Models are Few-Shot Learners
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• In-Context Learning Paper List

Instruction Tuning

• Finetuned Language Models Are Zero-Shot Learners
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Scaling Instruction-Finetuned Language Models
• Training Language Models to Follow Instructions with Human Feedback
• Self-Instruct: Aligning Language Model with Self Generated Instructions

Chain of Thoughts

Chain of Thought Prompting Elicits Reasoning in Large Language Models

Parameter-Efficient Fine-Tuning (PEFT)

• LoRA: Low-Rank Adaptation of Large Language Models
• Parameter-Efficient Transfer Learning for NLP
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• GPT Understands, Too
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models

LLM Inference, Benchmark and Development Tools

• llama.cpp
• MLC-LLM
• FastChat
• LangChain
• Huggingface
• Gradio
• vllm
• A Survey on Evaluation of Large Language Models