[Request] Please generate CodeLlama-7b-Python-hf-q4f16_1-metal.so

[Shekhars]

It's missing in the latest update. Can you please add this so that M1/M2 can run the codellama? Thanks

[sangelone]

The 13B would also be helpful 😄

[Sing-Li]

And for vulkan.dll please - many users are on Windows wanting to try CodeLlama 🙏

[sangelone]

Not sure if it fully resolves this request, but I at least built and added CodeLlama-13b-Python-hf-q4f16_1-metal in a PR: #18

[Sing-Li]

Thanks @sangelone ! How did you do it? (which instructions did you follow and/or did you have to modify any config/source files?) Some of us may be able to replicate it and help build some of the other missing targets. 🙏

[sangelone]

I just followed the docs using the appropriate model weights from HF on an M2 machine. It took a few hours to get all the dependencies installed and working in a Conda env, but then it was pretty straightforward to do the build itself, maybe half an hour more at that point.

[Sing-Li]

Thanks @sangelone ! Got the Windows DLL for 13b done. The 34b model's raw weights are 300GB in size 😱 Don't have enough fast NvME to work on it 😞