forked from ggerganov/llama.cpp
-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
2 changed files
with
111 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| ## MiniCPM-V 2.6 | ||
|
|
||
| ### Prepare models and code | ||
|
|
||
| Download [MiniCPM-V-2_6](https://huggingface.co/openbmb/MiniCPM-V-2_6) PyTorch model from huggingface to "MiniCPM-V-2_6" folder. | ||
|
|
||
| Clone llama.cpp: | ||
| ```bash | ||
| git clone https://github.com/ggerganov/llama.cpp | ||
| cd llama.cpp | ||
| ``` | ||
|
|
||
| ### Usage of MiniCPM-V 2.6 | ||
|
|
||
| Convert PyTorch model to gguf files (You can also download the converted [gguf](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) by us) | ||
|
|
||
| ```bash | ||
| python ./examples/llava/minicpmv-convert/minicpmv2_6-surgery.py -m ../MiniCPM-V-2_6 | ||
| python ./examples/llava/minicpmv-convert/minicpmv2_6-convert-image-encoder-to-gguf.py -m ../MiniCPM-V-2_6 --minicpmv-projector ../MiniCPM-V-2_6/minicpmv.projector --output-dir ../MiniCPM-V-2_6/ --image-mean 0.5 0.5 0.5 --image-std 0.5 0.5 0.5 | ||
| python ./convert-hf-to-gguf.py ../MiniCPM-V-2_6/model | ||
|
|
||
| # quantize int4 version | ||
| ./llama-quantize ../MiniCPM-V-2_6/model/ggml-model-f16.gguf ../MiniCPM-V-2_6/model/ggml-model-Q4_K_M.gguf Q4_K_M | ||
| ``` | ||
|
|
||
| Build for Linux or Mac | ||
|
|
||
| ```bash | ||
| make | ||
| make llama-minicpmv-cli | ||
| ``` | ||
|
|
||
| Inference on Linux or Mac | ||
| ``` | ||
| # run f16 version | ||
| ./llama-minicpmv-cli -m ../MiniCPM-V-2_6/model/ggml-model-f16.gguf --mmproj ../MiniCPM-V-2_6/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg -p "What is in the image?" | ||
| # run quantized int4 version | ||
| ./llama-minicpmv-cli -m ../MiniCPM-V-2_6/model/ggml-model-Q4_K_M.gguf --mmproj ../MiniCPM-V-2_6/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg -p "What is in the image?" | ||
| # or run in interactive mode | ||
| ./llama-minicpmv-cli -m ../MiniCPM-V-2_6/model/ggml-model-Q4_K_M.gguf --mmproj ../MiniCPM-V-2_6/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg -i | ||
| ``` | ||
|
|
||
| ### Video | ||
| Install FFmpeg | ||
| ``` | ||
| brew install ffmpeg | ||
| brew install pkg-config | ||
| ``` | ||
|
|
||
| ### Android | ||
|
|
||
| #### Build on Android device using Termux | ||
| We found that build on Android device would bring better runtime performance, so we recommend to build on device. | ||
|
|
||
| [Termux](https://github.com/termux/termux-app#installation) is a terminal app on Android device (no root required). | ||
|
|
||
| Install tools in Termux: | ||
| ``` | ||
| apt update && apt upgrade -y | ||
| apt install git make cmake | ||
| ``` | ||
|
|
||
| It's recommended to move your model inside the `~/` directory for best performance: | ||
| ``` | ||
| cd storage/downloads | ||
| mv model.gguf ~/ | ||
| ``` | ||
|
|
||
| #### Building the Project using Android NDK | ||
| Obtain the [Android NDK](https://developer.android.com/ndk) and then build with CMake. | ||
|
|
||
| Execute the following commands on your computer to avoid downloading the NDK to your mobile. Alternatively, you can also do this in Termux: | ||
|
|
||
| ```bash | ||
| mkdir build-android | ||
| cd build-android | ||
| export NDK=/your_ndk_path | ||
| cmake -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-23 -DCMAKE_C_FLAGS=-march=armv8.4a+dotprod .. | ||
| make | ||
| ``` | ||
|
|
||
| Install [termux](https://github.com/termux/termux-app#installation) on your device and run `termux-setup-storage` to get access to your SD card (if Android 11+ then run the command twice). | ||
|
|
||
| Finally, copy these built `llama` binaries and the model file to your device storage. Because the file permissions in the Android sdcard cannot be changed, you can copy the executable files to the `/data/data/com.termux/files/home/bin` path, and then execute the following commands in Termux to add executable permission: | ||
|
|
||
| (Assumed that you have pushed the built executable files to the /sdcard/llama.cpp/bin path using `adb push`) | ||
| ``` | ||
| $cp -r /sdcard/llama.cpp/bin /data/data/com.termux/files/home/ | ||
| $cd /data/data/com.termux/files/home/bin | ||
| $chmod +x ./* | ||
| ``` | ||
|
|
||
| Download models and push them to `/sdcard/llama.cpp/`, then move it to `/data/data/com.termux/files/home/model/` | ||
|
|
||
| ``` | ||
| $mv /sdcard/llama.cpp/ggml-model-Q4_K_M.gguf /data/data/com.termux/files/home/model/ | ||
| $mv /sdcard/llama.cpp/mmproj-model-f16.gguf /data/data/com.termux/files/home/model/ | ||
| ``` | ||
|
|
||
| Now, you can start chatting: | ||
| ``` | ||
| $cd /data/data/com.termux/files/home/bin | ||
| $./llama-minicpmv-cli -m ../model/ggml-model-Q4_K_M.gguf --mmproj ../model/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg -p "What is in the image?" | ||
| ``` |