Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CodeLlama-13b-Python-hf-q4f16_1-metal #18

Merged
merged 1 commit into from
Aug 30, 2023

Conversation

sangelone
Copy link
Contributor

@sangelone sangelone commented Aug 28, 2023

Addresses (in part) #16. Total param size is 6.8GB.

Build output:

Downloaded weights to dist/models/CodeLlama-13b-Python-hf
Using path "dist/models/CodeLlama-13b-Python-hf" for model "CodeLlama-13b-Python-hf"
/Users/catalyst/Workspace/mlc-ai-package-self-runner/_work/package/package/tvm/src/runtime/metal/metal_device_api.mm:167: Intializing Metal device 0, name=Apple M2
Host CPU dection:
  Target triple: arm64-apple-darwin22.6.0
  Process triple: arm64-apple-darwin22.6.0
  Host CPU: apple-m1
Target configured: metal -keys=metal,gpu -max_function_args=31 -max_num_threads=256 -max_shared_memory_per_block=32768 -max_threads_per_block=1024 -thread_warp_size=32
Host CPU dection:
  Target triple: arm64-apple-darwin22.6.0
  Process triple: arm64-apple-darwin22.6.0
  Host CPU: apple-m1
Automatically using target for weight quantization: metal -keys=metal,gpu -max_function_args=31 -max_num_threads=256 -max_shared_memory_per_block=32768 -max_threads_per_block=1024 -thread_warp_size=32
Start computing and quantizing weights... This may take a while.
Finish computing and quantizing weights.
Total param size: 6.820138931274414 GB
Start storing to cache dist/CodeLlama-13b-Python-hf-q4f16_1/params
All finished, 163 total shards committed, record saved to dist/CodeLlama-13b-Python-hf-q4f16_1/params/ndarray-cache.json
Finish exporting chat config to dist/CodeLlama-13b-Python-hf-q4f16_1/params/mlc-chat-config.json
Save a cached module to dist/CodeLlama-13b-Python-hf-q4f16_1/mod_cache_before_build.pkl.
Finish exporting to dist/CodeLlama-13b-Python-hf-q4f16_1/CodeLlama-13b-Python-hf-q4f16_1-metal.so

@Sing-Li
Copy link
Contributor

Sing-Li commented Aug 30, 2023

@CharlieFRuan is it possible to merge this 🙏

@MasterJH5574 MasterJH5574 merged commit 74a341e into mlc-ai:main Aug 30, 2023
@MasterJH5574
Copy link
Member

Merged. Thanks @sangelone!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants