Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Batch=32 / paged attention / and new RoPE module support to all Llama3 demo and tests #15327

Draft
wants to merge 28 commits into
base: main
Choose a base branch
from
Draft

Add Batch=32 / paged attention / and new RoPE module support to all Llama3 demo and tests #15327

wants to merge 28 commits into from
+1,588 −508

Commits on Nov 26, 2024

  1. #13368: Add rope module to llama3 codebase

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    722b71a View commit details
    Browse the repository at this point in the history

  2. #13368: Add paged attention to test llama attention. Update max batch… 

    … size to 32
    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    fa7cd65 View commit details
    Browse the repository at this point in the history

  3. #13368: Fix KV cache file to be replicated instead of sharded

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    3fd83f8 View commit details
    Browse the repository at this point in the history

  4. #13368: Add paged attention support and batch=32 support for test dec… 

    …oder
    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    0328db6 View commit details
    Browse the repository at this point in the history

  5. #13368: Add page attention and batch=32 support to test model. TODO i… 

    …nvestigate PCC reduction with batch>1
    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    a7d3d72 View commit details
    Browse the repository at this point in the history

  6. #13368: Addedrope and paged attn support to llama demo. TODO: Check b… 

    …ad output
    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    08fce34 View commit details
    Browse the repository at this point in the history

  7. #0: Add llama rope

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    2e38fb8 View commit details
    Browse the repository at this point in the history

  8. #0: Fix llama demo with batch size > 1 and paged attn. TODO: code cle… 

    …anup!
    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    4c13276 View commit details
    Browse the repository at this point in the history

  9. #0: Fixed the llama tests: attn, attn-prefill, decoder-prefill, decod… 

    …er, model
    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    564fb20 View commit details
    Browse the repository at this point in the history

  10. #0: Fix test llama model prefill

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    135df39 View commit details
    Browse the repository at this point in the history

  11. #0: Relax PCC check for test_llama_accuracy

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    ba73d3c View commit details
    Browse the repository at this point in the history

  12. #0: All llama tests now compatible with paged attention and llama rope

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    0165ddb View commit details
    Browse the repository at this point in the history

  13. #0: Add support for batch sizes that are not divisible by tile size, … 

    …and batch sizes that don't lead to a rectangular core grid. TODO: confirm if tracing works.
    @avoraTT @mtairum
    avoraTT authored and mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    b4f52a3 View commit details
    Browse the repository at this point in the history

  14. #0: Fix assert

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    85e0155 View commit details
    Browse the repository at this point in the history

  15. #0: use ttnn.argmax multicore for 1 user

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    57f6fc9 View commit details
    Browse the repository at this point in the history

  16. #0: [REVERT] Added mayo input

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    7d7536d View commit details
    Browse the repository at this point in the history

  17. #0: Remove debug code to speed up demo

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    5cca54d View commit details
    Browse the repository at this point in the history

  18. #0: Update debug max seqlen

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    a967c69 View commit details
    Browse the repository at this point in the history

  19. Add padding to position ids to support rope with batch < 32 in trace … 

    …mode. TODO: Debug inconsistent outputs of batch 1 vs batch 16/32
    @avoraTT @mtairum
    avoraTT authored and mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    03c2e79 View commit details
    Browse the repository at this point in the history

  20. #0: Fix llama rope on-host device for single-chip

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    c71c207 View commit details
    Browse the repository at this point in the history

  21. #0: Refactor llama3 test_attention

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    0ec756b View commit details
    Browse the repository at this point in the history

  22. #0: Fix N150-8B demo. Minor fixes after rebase.

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    4ff627a View commit details
    Browse the repository at this point in the history

  23. #0: Remove sliding_window references from llama3 codebase

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    29770af View commit details
    Browse the repository at this point in the history

  24. #0: Remove references to kv_seq_len to simplify llama3 codebase

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    a1ad346 View commit details
    Browse the repository at this point in the history

  25. Update rope to do padding internally. Add comments explaining inconsi… 

    …stency in output across batch sizes.
    @avoraTT
    avoraTT committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    9616567 View commit details
    Browse the repository at this point in the history

  26. Add fix for accuracy test to work for batch > 1

    @avoraTT
    avoraTT committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    834fecd View commit details
    Browse the repository at this point in the history

  27. #0: Refactored all llama3 tests and demo code

    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    a7f633c View commit details
    Browse the repository at this point in the history

  28. Merge branch 'mtairum/paged_attn_llama3' of https://github.com/tensto… 

    …rrent/tt-metal into mtairum/paged_attn_llama3

> Conflicts:
>	models/demos/llama3/demo/demo.py
    @mtairum
    mtairum committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    b15f45d View commit details
    Browse the repository at this point in the history