代码

在使用如下代码，进行大模型推理时，出现了报错，无法从本地路径导入大模型；

import os
import json
import pickle
import pandas as pd
from vllm import LLM, SamplingParams

sampling_params = SamplingParams(temperature=0, n=1, max_tokens=128)

# 大模型名字或本地路径
model_path = '../../../models/summary_cls'
# model_path = "/mnt/workspace/LLaMA-Factory/models/summary_cls"
llm = LLM(
    model=model_path,
    trust_remote_code=True,
    tokenizer=model_path,
    tokenizer_mode='auto',
    tensor_parallel_size=1
)

model_path 是本地经过lora权重合并后的大模型路径，无论是使用相对路径还是绝对路径都会报错；

使用相对路径报了如下错误：

return complexjson.loads(self.text, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/simplejson/__init__.py", line 514, in loads
    return _default_decoder.decode(s)
  File "/opt/conda/lib/python3.10/site-packages/simplejson/decoder.py", line 386, in decode
    obj, end = self.raw_decode(s)
  File "/opt/conda/lib/python3.10/site-packages/simplejson/decoder.py", line 416, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/workspace/LLaMA-Factory/demos/guihua_classify/scripts/test.py", line 13, in <module>
    llm = LLM(
  File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 109, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 351, in from_engine_args
    engine_configs = engine_args.create_engine_configs()
  File "/opt/conda/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 272, in create_engine_configs
    model_config = ModelConfig(self.model, self.tokenizer,
  File "/opt/conda/lib/python3.10/site-packages/vllm/config.py", line 96, in __init__
    model_path = snapshot_download(model_id=model,
  File "/opt/conda/lib/python3.10/site-packages/modelscope/hub/snapshot_download.py", line 98, in snapshot_download
    revision_detail = _api.get_valid_revision_detail(
  File "/opt/conda/lib/python3.10/site-packages/modelscope/hub/api.py", line 497, in get_valid_revision_detail
    all_branches_detail, all_tags_detail = self.get_model_branches_and_tags_details(
  File "/opt/conda/lib/python3.10/site-packages/modelscope/hub/api.py", line 578, in get_model_branches_and_tags_details
    d = r.json()
  File "/opt/conda/lib/python3.10/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

我当前的 vllm 版本如下：

Name: vllm
Version: 0.3.0
Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
Home-page: https://github.com/vllm-project/vllm
Author: vLLM Team
Author-email: 
License: Apache 2.0
Location: /opt/conda/lib/python3.10/site-packages
Requires: aioprometheus, fastapi, ninja, numpy, psutil, pydantic, pynvml, ray, sentencepiece, torch, transformers, uvicorn, xformers

解决方法

pip install vllm==0.3.3

升级一下 vllm 的版本就好了

不要无脑安装 vllm==0.3.3，使用最新的版本当然更好。