“This is the 8th day of my participation in the Gwen Challenge in November. Check out the details: The Last Gwen Challenge in 2021”
High reuse Bert model text classification code
High reuse Bert model text classification code (a) data reading
The source code interpretation
The models in the source code are stored separately in the Model folder. Take a look at module.py, which houses a simple fully connected neural network model as a classifier.
The network structure of classifier is very simple and consists of only two layers.
- Dropout layer
- Linear layer
# module.py
import torch.nn as nn
# classifier
class IntentClassifier(nn.Module) :
def __init__(self, input_dim, num_labels, dropout_rate=0.) :
super(IntentClassifier, self).__init__()
self.dropout = nn.Dropout(dropout_rate)
self.linear = nn.Linear(input_dim, num_labels)
def forward(self, x) :
x = self.dropout(x)
return self.linear(x)
Copy the codeLet’s focus on the Bert model code
import torch
import torch.nn as nn
from transformers import BertPreTrainedModel, BertModel, BertConfig
from torchcrf import CRF
from .module import IntentClassifier
class ClsBERT(BertPreTrainedModel) :
def __init__(self, config, args, label_lst) :
super(ClsBERT, self).__init__(config)
self.args = args
self.num_labels = len(label_lst)
self.bert = BertModel(config=config) # Load pretrained bert
self.classifier = IntentClassifier(config.hidden_size, self.num_labels, args.dropout_rate)
def forward(self, input_ids, attention_mask, token_type_ids, label_ids) :
outputs = self.bert(input_ids, attention_mask=attention_mask,
token_type_ids=token_type_ids) # sequence_output, pooled_output, (hidden_states), (attentions)
sequence_output = outputs[0]
pooled_output = outputs[1] # [CLS]
logits = self.classifier(pooled_output)
outputs = ((logits),) + outputs[2:] # add hidden states and attention if they are here
# 1. Intent Softmax
if label_ids is not None:
if self.num_labels == 1:
loss_fct = nn.MSELoss()
loss = loss_fct(logits.view(-1), label_ids.view(-1))
else:
loss_fct = nn.CrossEntropyLoss()
loss = loss_fct(logits.view(-1, self.num_labels), label_ids.view(-1))
outputs = (loss,) + outputs
return outputs # (loss), logits, (hidden_states), (attentions)
Copy the codeThe emphasis is on the forward part
outputs = self.bert(input_ids, attention_mask=attention_mask,
token_type_ids=token_type_ids) # sequence_output, pooled_output, (hidden_states), (attentions)
sequence_output = outputs[0] # sequence_output = outputs.last_hidden_state
pooled_output = outputs[1] # [CLS] / pooled_output = outputs.pooler_output
Copy the codeBert (input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids) returns two values Sequence_output, whose shape is (batch_size, bert_hidden_size), pooled_output, where pooler_output refers to the last hidden layer of the output sequence, i.e. CLS tag. Its shape size is (batch_size, bert_hidden_size)
- Can be achieved by
outputs[0]oroutputs.last_hidden_stateachievesequence_outputVector. - Can be achieved by
outputs[1]oroutputs.pooler_outputachievepooled_outputVector.
Generally, for classification tasks, the output of the last Bert layer is taken as the average pooling access linear layer, and the code can directly use outputs. Pooler_output as linear input. Elsider.last_hidden_state. mean(dim=1) can also be used as linear input. It is better to test the latter yourself.
Improved Bert output
We know that Bert model is composed of 12 transformer layers. What if we want to take out vectors of one layer or do vector stitching?
The output value of BertModel(BertPreTrainedModel) is the output value of BertPreTrainedModel.
Outputs: Tuple comprising various elements depending on the configuration (config) and inputs:
last_hidden_state: torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)
Sequence of hidden-states at the output of the last layer of the model.
pooler_output: torch.FloatTensor of shape (batch_size, hidden_size) Last layer hidden-state of the first token of the sequence (classification token)further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during Bert pretraining. This output is usually not a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence.
hidden_states: (optional, returned when config.output_hidden_states=True),list of torch.FloatTensor (one for the output of each layer + the output of the embeddings)of shape (batch_size, sequence_length, hidden_size): Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions: (optional, returned when config.output_attentions=True),list of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length):Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
There is a hidden_states layer that returns all layer vectors with shapes (batch_size, sequence_length, hidden_size). To output this list, config.output_hidden_states=True is configured during Bert initialization before hidden_states is returned
Let’s rewrite the code and try to extract the output vector of Bert’s third-to-last Transformer
class ClsBERT(BertPreTrainedModel) :
def __init__(self, config, args, label_lst) :
super(ClsBERT, self).__init__(config)
self.args = args
self.num_labels = len(label_lst)
self.bert = BertModel(config=config) # Load pretrained bert
self.classifier = IntentClassifier(config.hidden_size, self.num_labels, args.dropout_rate)
def forward(self, input_ids, attention_mask, token_type_ids, label_ids) :
""" Add output_hidden_states = True ""
outputs = self.bert(input_ids, attention_mask=attention_mask,token_type_ids=token_type_ids,output_hidden_states = True)
Pooled_outputs hidden_states[-3] mean(dim=1) Mean (batch_size,hidden_layers)
pooled_output = outputs.hidden_states[-3].mean(dim=1)
logits = self.classifier(pooled_output)
outputs = ((logits),) + outputs[2:] # add hidden states and attention if they are here
# 1. Intent Softmax
if label_ids is not None:
if self.num_labels == 1:
loss_fct = nn.MSELoss()
loss = loss_fct(logits.view(-1), label_ids.view(-1))
else:
loss_fct = nn.CrossEntropyLoss()
loss = loss_fct(logits.view(-1, self.num_labels), label_ids.view(-1))
outputs = (loss,) + outputs
return outputs # (loss), logits, (hidden_states), (attentions)
Copy the codeTensor: Torch. Empty (0, dtype=torch. Long). To (self.device) Use loop and cat to interpolate vector Change the input layer size of linear to the hidden_size of pooled_output
class ClsBERT(BertPreTrainedModel) :
def __init__(self, config, args, label_lst) :
super(ClsBERT, self).__init__(config)
self.args = args
self.num_labels = len(label_lst)
self.bert = BertModel(config=config) # Load pretrained bert
self.classifier = IntentClassifier(config.hidden_size, self.num_labels, args.dropout_rate)
def forward(self, input_ids, attention_mask, token_type_ids, label_ids) :
""" Add output_hidden_states = True ""
outputs = self.bert(input_ids, attention_mask=attention_mask,token_type_ids=token_type_ids,output_hidden_states = True)
State_input torch. Empty (0, Dtype =torch. Long).to(self.device) hidden_states[-3] mean(dim=1) Hidden_states (batch_size,hidden_layers)
pooled_output = torch.empty(0, dtype=torch.long).to(self.device)
for layer in outputs.hidden_states[self.concatnum:]:
pooled_output = torch.cat((pooled_output, layer.mean(dim=1)), dim=1)
""" modify end """
logits = self.classifier(pooled_output)
outputs = ((logits),) + outputs[2:] # add hidden states and attention if they are here
# 1. Intent Softmax
if label_ids is not None:
if self.num_labels == 1:
loss_fct = nn.MSELoss()
loss = loss_fct(logits.view(-1), label_ids.view(-1))
else:
loss_fct = nn.CrossEntropyLoss()
loss = loss_fct(logits.view(-1, self.num_labels), label_ids.view(-1))
outputs = (loss,) + outputs
return outputs # (loss), logits, (hidden_states), (attentions)
Copy the codeSo far, the explanation of model code is over, and we have also discussed and improved the output form of Bert model, including optimizer, learning rate and loss function, which will be explained in the model training code in the next article. NLP cute new, shallow talent, mistakes or imperfect place, please criticize!!