ChatGLM-6B模型导入问题

[Alternate-D]

作者您好！当我在前端输入框输入文本后，ChatGLM-6B模型推理时存在下面的问题：

INFO: Application startup complete.
INFO: 127.0.0.1:33704 - "OPTIONS / HTTP/1.1" 200 OK
Start generating...
The dtype of attention mask (torch.int64) is not bool
Traceback (most recent call last):
File "/media/ln01/2t/usr/cw/ChatZoo/generator/chatbot.py", line 64, in chat
output = self.generate(input_dict, gen_kwargs)
File "/media/ln01/2t/usr/cw/ChatZoo/generator/transformersbot.py", line 69, in generate
return self.model.generate(**input_dict, **gen_kwargs)
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/transformers/generation/utils.py", line 1485, in generate
return self.sample(
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/transformers/generation/utils.py", line 2524, in sample
outputs = self(
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/media/ln01/2t/usr/cw/ChatZoo/generator/models/chatglm/modeling_chatglm.py", line 1178, in forward
transformer_outputs = self.transformer(
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/ln01/2t/usr/cw/ChatZoo/generator/models/chatglm/modeling_chatglm.py", line 984, in forward
layer_ret = layer(
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/media/ln01/2t/usr/cw/ChatZoo/generator/models/chatglm/modeling_chatglm.py", line 624, in forward
attention_input = self.input_layernorm(hidden_states)
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 190, in forward
return F.layer_norm(
File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/functional.py", line 2515, in layer_norm
return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: expected scalar type Half but found Float

请问您能帮助我吗？如果可以的话感激不尽！

[Alternate-D]

还有一点，我想知道你们的具体虚拟环境，麻烦您告诉我

[x54-729]

请问您是使用的最新的代码吗？chatglm的模型内部调用了Tensor.half()接口，所以对输入的类型有邀请，可以尝试一下启动时指定 --dtype=float32

[Alternate-D]

请问您是使用的最新的代码吗？chatglm的模型内部调用了Tensor.half()接口，所以对输入的类型有邀请，可以尝试一下启动时指定 --dtype=float32

1.我使用的是您的最新代码；
2.您的server.py文件中dtype默认就是float32
parser.add_argument(
"--dtype", type=str, default="float32",
help="Dtype to load model."
)
所以还是存在上述问题的

[Alternate-D]

请问您是使用的最新的代码吗？chatglm的模型内部调用了Tensor.half()接口，所以对输入的类型有邀请，可以尝试一下启动时指定 --dtype=float32

我已经解决这个问题了！根据您的提示，我在_transformerbot.py_的_load_model(self)_方法下进行了如下修改：
_def load_model(self):
"""
Load model through transformers.
"""
# mute warning
trust_remote_code = issubclass(
self.model_cls, BaseAutoModelClass
)
# print(self.model_cls)
# print(type(self.model_cls))
# print(self.model_cls == ChatGLMForConditionalGeneration)
self.model = self.model_cls.from_pretrained(
self.model_name, torch_dtype=self.config.dtype,
device_map="auto", trust_remote_code=trust_remote_code
)
if (self.model_cls == ChatGLMForConditionalGeneration):
self.model = self.model.half()
即：当导入的模型为ChatGLM时，手动把数据类型转换为half，希望您可以参考。
最后，感谢您百忙中抽出时间为我解答！

[x54-729]

请问您是使用的最新的代码吗？chatglm的模型内部调用了Tensor.half()接口，所以对输入的类型有邀请，可以尝试一下启动时指定 --dtype=float32

1.我使用的是您的最新代码； 2.您的server.py文件中dtype默认就是float32 parser.add_argument( "--dtype", type=str, default="float32", help="Dtype to load model." ) 所以还是存在上述问题的

不好意思一开始回复错了，应该是--dtype=float16 _(:з不过chatglm里存在一个assert语句，用float32应该会报错才对