Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

llama: implement YaRN RoPE scaling #2268

Merged
merged 36 commits into from
Nov 1, 2023
Merged

llama: implement YaRN RoPE scaling #2268

merged 36 commits into from
Nov 1, 2023

Commits on Aug 7, 2023

  1. llama: implement NTK-By-Parts (NTKv2) RoPE scaling

    @cebtenzzre
    cebtenzzre committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    8dec38c View commit details
    Browse the repository at this point in the history

  2. CUDA implementation

    @cebtenzzre
    cebtenzzre committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    6aeb46b View commit details
    Browse the repository at this point in the history

  3. Metal implementation

    @cebtenzzre
    cebtenzzre committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    9348aa4 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2023

  1. implement new YaRN algorithm

    @cebtenzzre
    cebtenzzre committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    a30ae20 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2023

  1. Merge branch 'master' of https://github.com/ggerganov/llama.cpp into … 

    …ntkv2-fix
    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    b5ced4f View commit details
    Browse the repository at this point in the history

  2. ggml : increase GGML_MAX_OP_PARAMS

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    826269a View commit details
    Browse the repository at this point in the history

  3. YaRN : avoid NaN if unused betas are zero

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    cf731d5 View commit details
    Browse the repository at this point in the history

  4. YaRN : fix missing parameter in CUDA impl

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    dcb058c View commit details
    Browse the repository at this point in the history

  5. convert : reduce unnecessary variables in Params

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    281b26e View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' of https://github.com/ggerganov/llama.cpp into … 

    …ntkv2
    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    a06c729 View commit details
    Browse the repository at this point in the history

  7. llama : simplify use of context params

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    dc26a0d View commit details
    Browse the repository at this point in the history

  8. llama : store YaRN parameters in GGUF

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    904d4ed View commit details
    Browse the repository at this point in the history

  9. fix convert scripts

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    56abb9a View commit details
    Browse the repository at this point in the history

  10. llama : fix C compatibility

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    43eaf06 View commit details
    Browse the repository at this point in the history

  11. don't hardcode max_pos_emb

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    fe788c4 View commit details
    Browse the repository at this point in the history

  12. address review comments

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    e0b120c View commit details
    Browse the repository at this point in the history

  13. restore backwards compatiblity with *.rope.scale_linear

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    19bb74e View commit details
    Browse the repository at this point in the history

  14. better option descriptions in help

    @cebtenzzre
    cebtenzzre committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    4d5fe73 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2023

  1. gguf : store scaling type as a string instead of an int

    @cebtenzzre
    cebtenzzre committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    7466415 View commit details
    Browse the repository at this point in the history

  2. improve printing of YaRN parameters

    @cebtenzzre
    cebtenzzre committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    4f4e948 View commit details
    Browse the repository at this point in the history

  3. allow forcing ext_factor to zero if scaling type is YaRN

    @cebtenzzre
    cebtenzzre committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    5d7a3a5 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' of https://github.com/ggerganov/llama.cpp into … 

    …ntkv2
    @cebtenzzre
    cebtenzzre committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    9bd050f View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2023

  1. fix rope_cuda parameter order

    @cebtenzzre
    cebtenzzre committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    babf0e0 View commit details
    Browse the repository at this point in the history

  2. default n_yarn_orig_ctx to n_ctx_train

    @cebtenzzre
    cebtenzzre committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    0050e1e View commit details
    Browse the repository at this point in the history

  3. fix uninitialized cparams

    @cebtenzzre
    cebtenzzre committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    09c3102 View commit details
    Browse the repository at this point in the history

  4. make printed param formatting more consistent

    @cebtenzzre
    cebtenzzre committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    57c3442 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. fix missing import

    @cebtenzzre
    cebtenzzre committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    a20b3e6 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2023

  1. Merge branch 'master' of https://github.com/ggerganov/llama.cpp into … 

    …ntkv2
    @cebtenzzre
    cebtenzzre committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    9ef91b1 View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2023

  1. Fix YaRN inverted scaling and add "rope.scaling.type" to GGUF (#1)

    @jquesnelle
    jquesnelle committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    9ae10b3 View commit details
    Browse the repository at this point in the history

  2. fix YaRN ramp, make mscale conditional, add --yarn-orig-ctx (#2)

    @jquesnelle
    jquesnelle committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    14cf93b View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2023

  1. Merge branch 'master' of https://github.com/ggerganov/llama.cpp into … 

    …ntkv2
    @cebtenzzre
    cebtenzzre committed Oct 22, 2023
    Configuration menu
    Copy the full SHA
    237f1e7 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2023

  1. Merge branch 'master' of https://github.com/ggerganov/llama.cpp into … 

    …ntkv2
    @cebtenzzre
    cebtenzzre committed Oct 23, 2023
    Configuration menu
    Copy the full SHA
    bc8395d View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2023

  1. Merge branch 'master' of https://github.com/ggerganov/llama.cpp into … 

    …ntkv2
    @cebtenzzre
    cebtenzzre committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    4d5ed83 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2023

  1. fix loading rope.scaling.original_context_length from GGUF (#3) 

    Co-authored-by: cebtenzzre <[email protected]>
    @jquesnelle @cebtenzzre
    jquesnelle and cebtenzzre committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    9fc8238 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2023

  1. implement YaRN for GPT-NeoX RoPE

    @cebtenzzre
    cebtenzzre committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    15f26ef View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/ggerganov/llama.cpp into … 

    …ntkv2
    @cebtenzzre
    cebtenzzre committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    081f738 View commit details
    Browse the repository at this point in the history