supposed Docker is installed.
# Powershell, use ${pwd} instead of %CD%
> docker run -it --rm --gpus all -v %CD%:/workspace pytorch/pytorch:2.4.0-cuda12.1-cudnn9-devel
# inside container
root@473c7bb9aa01:/workspace# pip install transformers accelerate
Before starting, need a hugging face ID and access to the repository.
Receive an access token with Read privileges in the Hugging Face's setting.
Token starts with hf_
root@473c7bb9aa01:/workspace# huggingface-cli login
~
To login, `huggingface_hub` requires a token generated from https://huggingface.co/settings/tokens .
Enter your token (input will not be visible): **TYPE TOKEN HERE**
Add token as git credential? (Y/n) n
~~
Login successful
At first time, it may take time to download related files.
Test it by changing the prompt in the python file (line 13:14).
root@473c7bb9aa01:/workspace# python quickstart.py
Note that it's inefficient to load the model every time.