Hugging Face's logo

ibm-granite
/
granite-4.0-h-small

Text Generation
Transformers
Safetensors
granitemoehybrid
language
granite-4.0
conversational
Model card Files
xet
Community
8

VLLM Support

#1
by jtvino - opened
Discussion
jtvino

Hi Granite Team, thanks for the contribution! Will this model get VLLM support?

gabegoodhart
 IBM Granite org

Hi @jtvino ! Yes, this model is fully supported in vLLM today

jtvino

Hi @jtvino ! Yes, this model is fully supported in vLLM today

Perfect! Is there any documentation on how to deploy it? E.g Preferred settings and HW compatibility?

gabegoodhart
 IBM Granite org

There's a high-level guide here: https://www.ibm.com/granite/docs/run/granite-with-vllm-containerized

It's a great idea to dig further into the specific configuration choices for the different model sizes and hardware options.

Sign up or log in to comment