IFU master 2024 01 24 #4
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Changed misleading Hardware requirements from "2018 or older" to "2018 or newer"
Added scipy to requirements.txt as it is used but not added to requirements
…yer-device Add `device` parameter to `Linear` subclasses and `Embedding`
Update README.md
Update README.md
Fix parameter name in error message
Add version attribute as per Python convention
…_permissionerror_order Make sure bitsandbytes handles permission errors in the right order
Added scipy to requirements.txt
Fix typo "quanitze"
fix array index out of bounds in kgetColRowStats
…sbelkada-delete-workflow Delete .github/workflows/delete_doc_commment.yml
This PR adds initial FSDP support for training QLoRA models. It enables basic FSDP and CPU Offload support, with low memory training via FSDP.sync_module_states option unsupported. This PR builds off of bitsandbytes-foundation#840 commit 8278fca and BNB FSDP by @TimDettmers and @Titus-von-Koeller. An example of using this PR to finetune QLoRA models with FSDP can be found in the demo repo: AnswerDotAi/fsdp_qlora. * Minimal changes for fp32 4bit storage from BNB commit 8278fca * Params4bit with selectable storage dtype * possible fix for double quantizing linear weight & quant storage dtype * minor fixes in Params4bit for peft tests * remove redundant * add float16 * update test * Remove float16 quant cast as there are fp32, bf16, & fp16 quant kernels --------- Co-authored-by: Kerem Turgutlu <keremturgutlu@gmail.com>
…ytes-foundation#703), Sort compute capabilities sets to select max * Add support for CUDA 12.1 * Update README to include CUDA 12.1 version * Add support for >= 12x Co-authored-by: Jeongseok Kang <jskang@lablup.com> * Temporary version of bitsandbytes PR 527: Sort compute capabilities sets to select max * Modify PR 506 to support C++20 * Add Cuda 12.2 --------- Co-authored-by: PriNova <info@prinova.de> Co-authored-by: PriNova <31413214+PriNova@users.noreply.github.com> Co-authored-by: Jeongseok Kang <jskang@lablup.com>
Remove redundant key
* Added install requirements to setup * Update setup.py Co-authored-by: Aarni Koskela <akx@iki.fi> --------- Co-authored-by: Aarni Koskela <akx@iki.fi>
* implicitly skip any test that implicitly uses CUDA on a non-CUDA box * add a `requires_cuda` fixture
pnunna93
requested review from
Lzy17,
amathews-amd,
dllehr-amd and
jpvillam-amd
amathews-amd
approved these changes
Jan 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR pulls upstream changes for 0.42.0 version.
Unit test summary:
PreIFU:
PostIFU: