Error when running model with vllm
#1
by
melching
- opened
Hello,
im currently trying to get this model running but always encounter an error when trying to do so with vllm (using the command you provided).
...
(EngineCore_0 pid=46840) ERROR 08-28 18:11:38 [core.py:700] File "/home/me/.venv/lib/python3.12/site-packages/vllm/model_executor/models/internvl.py", line 1061, in __init__
(EngineCore_0 pid=46840) ERROR 08-28 18:11:38 [core.py:700] self.language_model.make_empty_intermediate_tensors)
(EngineCore_0 pid=46840) ERROR 08-28 18:11:38 [core.py:700] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=46840) ERROR 08-28 18:11:38 [core.py:700] File "/home/me/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1940, in __getattr__
(EngineCore_0 pid=46840) ERROR 08-28 18:11:38 [core.py:700] raise AttributeError(
(EngineCore_0 pid=46840) ERROR 08-28 18:11:38 [core.py:700] AttributeError: 'GptOssForCausalLM' object has no attribute 'make_empty_intermediate_tensors'
These are my currently installed version:
transformers==4.55.4
torch==2.7.1
vllm==0.10.1.1
Do you have an Idea why I'm getting this error? I assume you got it running.
Thanks
Hmm, I'll need to dig a little into the Gpt-oss model in the vllm project. In fairness, I tried a handful of the qwen based internvl 3.5 models but didn't try this one specifically. Given this is a different based model from the qwen ones, it makes sense this one may not "just work". Will try to dig in this week