Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cog 417 chunking unit tests #205

Merged
merged 42 commits into from
Nov 18, 2024
Merged

Cog 417 chunking unit tests #205

merged 42 commits into from
Nov 18, 2024

Commits on Nov 13, 2024

  1. Small cosmetic changes

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    6f0637a View commit details
    Browse the repository at this point in the history

  2. Add isomorphism tests

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    98cbaaf View commit details
    Browse the repository at this point in the history

  3. Fix chunk_by_word_test

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    830c671 View commit details
    Browse the repository at this point in the history

  4. Make chunk_by_word isomorphic

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    c054e89 View commit details
    Browse the repository at this point in the history

  5. Adapt chunk_by_sentence to isomorphic chunk_by_word

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    ab55a73 View commit details
    Browse the repository at this point in the history

  6. Refactor chunk_by_paragraph to be isomorphic

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    ce498d9 View commit details
    Browse the repository at this point in the history

  7. Autoformat chunking tests

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    92a66dd View commit details
    Browse the repository at this point in the history

  8. Adapt chunk_by_paragraph test parametrization

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    ef7a190 View commit details
    Browse the repository at this point in the history

  9. Add maximum_length argument to chunk_sentences

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    f8e5b52 View commit details
    Browse the repository at this point in the history

  10. Adapt chunk_by_paragraph_test.py

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    1b4a7e4 View commit details
    Browse the repository at this point in the history

  11. Fix PdfDocument teset, give chunk_by_sentence a maximum_length arg

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    9b2fb09 View commit details
    Browse the repository at this point in the history

  12. Replace word_count with maximum_length in if clause

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    9ea2634 View commit details
    Browse the repository at this point in the history

  13. Test maximum_lenth parameter of chunk_by_sentence

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    fdec9a6 View commit details
    Browse the repository at this point in the history

  14. Add more adversarial examples

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    b787407 View commit details
    Browse the repository at this point in the history

  15. Remove assert and move is_real_paragraph_end outside loop

    @0xideas
    0xideas committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    45a60b7 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. Simplify chunk_by_word

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    d906983 View commit details
    Browse the repository at this point in the history

  2. Add ellipsis test string

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    adc8a0b View commit details
    Browse the repository at this point in the history

  3. Return stripped value from get_embeddable_data if its string

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    e794bb8 View commit details
    Browse the repository at this point in the history

  4. Move PdfDocument_test.py to integration tests

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    8afb25e View commit details
    Browse the repository at this point in the history

  5. Add TextDocument_test.py

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    e663675 View commit details
    Browse the repository at this point in the history

  6. Change test_input order

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    c905510 View commit details
    Browse the repository at this point in the history

  7. Add transcribe_image and create_transcript methods

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    8b3b2f8 View commit details
    Browse the repository at this point in the history

  8. Fix lambda bug in AudioDocument and ImageDocument

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    f87fd12 View commit details
    Browse the repository at this point in the history

  9. Add AudioDocument and ImageDocument tests

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    8260647 View commit details
    Browse the repository at this point in the history

  10. Update cognee/tasks/chunks/chunk_by_word.py 

    Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
    @0xideas @coderabbitai
    0xideas and coderabbitai[bot] authored Nov 14, 2024
    Configuration menu
    Copy the full SHA
    f2206a0 View commit details
    Browse the repository at this point in the history

  11. Fix chunk_index bug in chunk_by_paragraph

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    6721eae View commit details
    Browse the repository at this point in the history

  12. Save paragraph_ids in chunk_by_paragraph

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    57d8149 View commit details
    Browse the repository at this point in the history

  13. Change chunk_by_word to collect newlines in prior words

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    eaf9167 View commit details
    Browse the repository at this point in the history

  14. Change document test ground truth values for new chunk_by_word

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    a52d3ac View commit details
    Browse the repository at this point in the history

  15. Set batch_paragraph=False in run_chunking_test

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    b4d509e View commit details
    Browse the repository at this point in the history

  16. Fix sentence_cut return value in inappropriate places

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    73f24f9 View commit details
    Browse the repository at this point in the history

  17. Update cognee/tasks/chunks/chunk_by_paragraph.py 

    Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
    @0xideas @coderabbitai
    0xideas and coderabbitai[bot] authored Nov 14, 2024
    Configuration menu
    Copy the full SHA
    8b68152 View commit details
    Browse the repository at this point in the history

  18. Return sentence_cut instead of word in chunk_by_paragraph

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    d6a6a9e View commit details
    Browse the repository at this point in the history

  19. Merge latest main

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    7cf8c74 View commit details
    Browse the repository at this point in the history

  20. Fix paragraph_ids handling

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    15420dd View commit details
    Browse the repository at this point in the history

  21. Remove chunk_index attribute from chunk_by_sentence return value

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    84c98f1 View commit details
    Browse the repository at this point in the history

  22. Test chunk_by_paragraph chunk numbering

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    928e107 View commit details
    Browse the repository at this point in the history

  23. Fix indexing in tests in chunk_by_sentence_test

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    14dd605 View commit details
    Browse the repository at this point in the history

  24. Refactor word_type yielding in chuck_by_sentence

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    e40e738 View commit details
    Browse the repository at this point in the history

  25. Remove unneeded document.read in AudioDocument_test

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    f51a44f View commit details
    Browse the repository at this point in the history

  26. Merge branch 'main' into COG-417-chunking-unit-tests

    @0xideas
    0xideas committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    6e416ed View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2024

  1. Merge branch 'main' into COG-417-chunking-unit-tests

    @borisarzentar
    borisarzentar authored Nov 17, 2024
    Configuration menu
    Copy the full SHA
    22a0e43 View commit details
    Browse the repository at this point in the history