诸神缄默不语-个人****博文目录
运行环境，警告信息和解决方案：
Linux环境，Python3，PyTorch版本为1.8.1，transformers包版本为4.12.5。
代码：
from transformers import AutoTokenizer,AutoModel
pretrained_path="mypath/bert-base-chinese"
tokenizer=AutoTokenizer.from_pretrained(pretrained_path)
encoder=AutoModel.from_pretrained(pretrained_path)
（模型文件下载自：bert-base-chinese · Hugging Face）
警告信息：
Some weights of the model checkpoint at mypath/bert-base-chinese were not used when initializing BertModel: ['cls.seq_relationship.weight', '', 'cls.seq_relationship.bias', '', '', '', '', '']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
这是个警告信息，不是报错信息。这只说明对应的加载的预训练模型与任务类型不完全对应。如这个模型的架构是BertForMaskedLM，因此用BERT在别的任务上的model类来调用该预训练模型时，要么出现有些参数用不到的情况（如用于MLM任务的预测头），要么出现有些参数没有、需要随机初始化的情况（如用于分类任务的预测头）。
本例由于我只想输出transformer模型的last hidden state，因此用不到警告信息中所说的这些分类参数。
如果你想直接删除这个信息，可以使用：
from transformers import logging
logging.set_verbosity_warning()
或：
from transformers import logging
logging.set_verbosity_error()
以下介绍一些相关的知识点：
理论上应该完全匹配（中给出的architectures就是BertForMaskedLM），但是仍会显示有些参数用不到：
from transformers import AutoTokenizer,BertForMaskedLM
pretrained_path="mypath/bert-base-chinese"
tokenizer=AutoTokenizer.from_pretrained(pretrained_path)
encoder=BertForMaskedLM.from_pretrained(pretrained_path)
警告信息：
Some weights of the model checkpoint at mypath/bert-base-chinese were not used when initializing BertForMaskedLM: ['cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
参考/huggingface/transformers/issues/5421#issuecomment-717245807，应该是因为官方的BERT模型有两个预测头（MLM和NSP），所以MLM任务上的模型没有加载NSP的预测头。
有新的参数需要随机初始化的情况（AutoModelForSequenceClassification这类属于需要在原始模型的基础上进行微调的模型类，对其进一步的了解可参考我撰写的另一篇博文：用在文本分类任务上微调预训练模型_诸神缄默不语的博客-****博客_huggingface transformers微调）：
from transformers import AutoConfig,AutoTokenizer,AutoModelForSequenceClassification
model_path="mypath/bert-base-chinese"
config=AutoConfig.from_pretrained(model_path,num_labels=5)
tokenizer=AutoTokenizer.from_pretrained(model_path)
encoder=AutoModelForSequenceClassification.from_pretrained(model_path,config=config)
警告信息：
Some weights of the model checkpoint at mypath/bert-base-chinese were not used when initializing BertForSequenceClassification: ['', '', '', 'cls.seq_relationship.bias', '', '', 'cls.seq_relationship.weight', '']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at mypath/bert-base-chinese and are newly initialized: ['', '']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
其他正文及脚注未提及的参考资料：
- nlp - Python: BERT Error - Some weights of the model checkpoint at were not used when initializing BertModel - Stack Overflow
- Huggingface-NLP笔记7：使用Trainer API来微调模型_javastart的博客-****博客
- What to do about this warning message: “Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification” · Issue #5421 · huggingface/transformers：文中提及了这个issue中的一个comment
- Token Classification - Colaboratory：介绍到了一点相关内容