Abstract: In the field of information structure extraction, the predecessors generally need to complete the information structure extraction based on manual annotation template. This paper proposes a zero-shot solution based on graph convolutional network, which can solve the problems where the training set and the test set come from different vertical fields.
This article is shared from Huawei Cloud community “Paper Interpretation Series 16: Information Structured Extraction in zero-Shot Scenarios” by the author: a smile.
In the field of information structured extraction, the predecessors usually need to complete the information structured extraction based on manual annotation template. This paper proposes a zero-shot solution based on graph convolutional network, which can solve the problems where the training set and the test set come from different vertical fields.
Figure 1. Different verticals for training and reasoning data sources.
Problem definition
Figure 2. Intuitive understanding of OpenIE and ClosedIE
Relatin Extraction
- Close Relation Extraction (ClasedIE)
R_R_ represents a collection of categories, including no categories. The model can directly assign categories to each entity.
- Open Relation Extraction(OpenIE)
R_R_ represents a set of categories. The model classifies two categories and determines whether one entity is the key of the other.
Zero-Shot Extraction
Zero-shots can be classified by difficulty as follows:
- Unseen-Website Zero-shot Extraction
That is, different formats in the same vertical, for example, a web page from a movie. It’s just that the reasoning test uses a different layout than the training.
- Unseen-Websiste Zero-shot Extraction
These are different formats for different verticals. For example, the training might be from a movie web page, while the reasoning test might be from a recruiting web page.
In fact, the solution proposed in this paper is to dig out all the key-value pairs in the graph network. Since the task of digging out key-value itself is independent of the layout, it plays a cross-domain layout structure analysis.
concept
-
Function: refers to the key
-
Object: refers to the value
-
Relationship: indicates key -> value
Encoder (Feature construction)
The construction of node information is done by the graph G_G_, which consists of a series of nodes N_N_ (entities) and Edges E (Edges) between the nodes.
Relationships between entities are constructed based on rules of design
Edges are built between nodes (key-value pairs are usually up and down or left and right) in the following cases:
-
Horizontal situation: horizontal neighbor, and no other nodes in the middle;
-
Vertical: The neighbor is vertical, and there is no other node in the middle;
-
Peer level: peer level nodes;
Use graph networks to model the relationships between entities
Node relationships are modeled based on Graph Attention Network (GAT). Initial (input) characteristics of nodes are as follows:
-
Visual features: visual description of nodes in web pages;
-
Text features: OpenIE is a feature average for pre-trained Bert, and CloseIE is a feature average for the frequency of occurrences of the node string (which seems more cross-domain friendly);
Pre-training mechanism
In this paper, an auxiliary loss function L_{pre}_Lpre_ is designed to supervise three categories: {key,value, other}. At the same time, in order to prevent overfitting in the training process, the graph network weight in the OpenIE task will not be updated after the pre-training is completed.
Relational prediction network
OpenIE
Key to determine whether a pair of nodes satisfies that the first node string contents are the second node string contents:
-
The candidate pair Identification algorithm is used to obtain potential string pairs;
-
The original input feature of the two nodes +GNN output feature + the relation feature of the two nodes are used as the input of the classifier.
-
Classification of fully connected networks;
ClosedIE
Cross entropy multi-class classification
The experiment
- It is true that cross-disciplinary tasks are more difficult.
- CloseIE: The more sites, the better.
- Identify the influence of various factors on the effect of the network model.
Click follow to learn about the fresh technologies of Huawei Cloud