Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SPFresh implementation based on SPTAG (#405) #406

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

MaggieQi
Copy link
Member

  • Initial commit

  • init from update version

  • merged code

  • backup

  • continue

  • update the initial work version

  • fix ExtraDynamicSearcher

  • fix vectorTranslateMap

  • fix compiling issues

  • fix compling

  • fix compiling

  • add merge

  • add merge re-assign

  • TODO: add steady state test

  • init steady state test

  • Init SPDKIO

  • optimize split

  • fix align_round

  • modify steady state: delete after search

  • modify trace

  • need to figure out disk usage

  • modify steadystate

  • need to debug load index

  • fix delete by metadata

  • add spfresh exe

  • finish benchmark

  • steady state complete

  • add KVTest

  • fix compiling errors

  • Init SPDKIO

  • optimize split

  • add KVTest

  • fix compiling errors

  • change update vector to partital read

  • modify merge

  • fix build

  • mem impl

  • fix correctness

  • Refactor code

  • add spdk as module

  • add isal-l_crypto as submodule

  • initial spdk integration

  • build queue infra

  • spdk working

  • add reference bdev.json

  • static spfresh search done

  • need to debug split

  • debug for GC/ReAssign

  • finish debug for GC/ReAssign

  • Add build guide for SPDK version

  • remove rocksdb

  • add timeout support for SPDK

  • modify code for timeout support

  • fix timeout bug

  • fix writeblocks segmentation fault

  • fix bug: SPDK init fail

  • fix timeout bugs

  • batch ReadBlocks

  • add batchSize option

  • add timeout support for batched I/O

  • Refine Index

  • debug refine

  • modify for stress test

  • change update from metadata to mapping

  • modify

  • add IOStatistics for SPDK

  • change ssd size

  • modify for stress test

  • fix ssd usage bug

  • fix io limit

  • modify ssd size for stress test

  • for io stat

  • modify for io

  • fix for stress test

  • temporarily remove rebuild

  • fix BKT bug

  • merge changes from Index.h BKTIndex.cpp and RelativeNeighborhoodGraph.h

  • add lock for mapping

  • fix merge bug

  • Improve README.md

  • add loading exclude head

  • modify length

  • fix code with AE repo

  • add checkpoint & move rocksdb to thirdparty

  • fix zstd dependency

  • fix compiling

  • fix running on SPDK

  • enable rocksdb, need to debug update

  • fix update

  • add SPDK & Rocksdb checkpoint function

  • pass test for checkpoint, todo: add persistentbuffer

  • add wal logic


* Initial commit

* init from update version

* merged code

* backup

* continue

* update the initial work version

* fix ExtraDynamicSearcher

* fix vectorTranslateMap

* fix compiling issues

* fix compling

* fix compiling

* add merge

* add merge re-assign

* TODO: add steady state test

* init steady state test

* Init SPDKIO

* optimize split

* fix align_round

* modify steady state: delete after search

* modify trace

* need to figure out disk usage

* modify steadystate

* need to debug load index

* fix delete by metadata

* add spfresh exe

* finish benchmark

* steady state complete

* add KVTest

* fix compiling errors

* Init SPDKIO

* optimize split

* add KVTest

* fix compiling errors

* change update vector to partital read

* modify merge

* fix build

* mem impl

* fix correctness

* Refactor code

* add spdk as module

* add isal-l_crypto as submodule

* initial spdk integration

* build queue infra

* spdk working

* add reference bdev.json

* static spfresh search done

* need to debug split

* debug for GC/ReAssign

* finish debug for GC/ReAssign

* Add build guide for SPDK version

* remove rocksdb

* add timeout support for SPDK

* modify code for timeout support

* fix timeout bug

* fix writeblocks segmentation fault

* fix bug: SPDK init fail

* fix timeout bugs

* batch ReadBlocks

* add batchSize option

* add timeout support for batched I/O

* Refine Index

* debug refine

* modify for stress test

* change update from metadata to mapping

* modify

* add IOStatistics for SPDK

* change ssd size

* modify for stress test

* fix ssd usage bug

* fix io limit

* modify ssd size for stress test

* for io stat

* modify for io

* fix for stress test

* temporarily remove rebuild

* fix BKT bug

* merge changes from Index.h BKTIndex.cpp
and RelativeNeighborhoodGraph.h

* add lock for mapping

* fix merge bug

* Improve README.md

* add loading exclude head

* modify length

* fix code with AE repo

* add checkpoint & move rocksdb to thirdparty

* fix zstd dependency

* fix compiling

* fix running on SPDK

* enable rocksdb, need to debug update

* fix update

* add SPDK & Rocksdb checkpoint function

* pass test for checkpoint, todo: add persistentbuffer

* add wal logic

---------

Co-authored-by: MaggieQi <chenqi871025@gmail.com>
Co-authored-by: cheqi <cheqi@microsoft.com>
Co-authored-by: Hengyu Liang <lhy1210302421@mail.ustc.edu.cn>
Co-authored-by: Ziyue Yang <yzylivezh@hotmail.com>
Co-authored-by: Hengyu Liang <lhy_ustc@mail.ustc.edu.cn>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants