moonshotai

/

Kimi-VL-A3B-Thinking-2506

like 295

Follow

Moonshot AI 2.77k

I am trying to host this on vLLM with Pipeline Parallelism as I have 2 GPUs on 2 different nodes. But looks like its not support for this model. I have loaded some other models and pipeline parallelism is working well.

stack trace:

(APIServer pid=35) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=35) return await anext(self.gen)
(APIServer pid=35) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=35) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 204, in build_async_engine_client_from_engine_args
(APIServer pid=35) vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=35) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=35) File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1335, in create_engine_config
(APIServer pid=35) config = VllmConfig(
(APIServer pid=35) ^^^^^^^^^^^
(APIServer pid=35) File "/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.py", line 120, in init
(APIServer pid=35) s.pydantic_validator.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
(APIServer pid=35) File "/usr/local/lib/python3.12/dist-packages/vllm/config/init.py", line 3998, in post_init
(APIServer pid=35) self.model_config.verify_with_parallel_config(self.parallel_config)
(APIServer pid=35) File "/usr/local/lib/python3.12/dist-packages/vllm/config/init.py", line 1318, in verify_with_parallel_config
(APIServer pid=35) raise NotImplementedError(
(APIServer pid=35) NotImplementedError: Pipeline parallelism is not supported for this model. Supported models implement the SupportsPP interface.

it seems to me the model supports pipeline parallel, maybe you just need to add a SupportsPP interface. Feel free to open a PR in vLLM.

Thank you!

It has been added in https://github.com/vllm-project/vllm/commit/fda9537c5e61ea8226fa7e5b60912deda97a8aab