Textual entailment

We consider a fundamental task in natural language processing: given two natural language sentences, a premise P and a hypothesis H, determine whether the premise entails, contradicts, or is neutral with respect to the given hypothesis. This is called textual entailment, or natural language inference (NLI).

This is a powerful idea, because if you have a system that can perform natural language inference, the system can take one sentence as input to determine what other sentences are implied by it. In other words, such a system can be said to be capable of loosely reasoning over text. This is fundamental to human language understanding and communication.

As an example, consider a system that answers questions about images with captions. An image is described with the following caption, which we’ll call the premise P: “A soccer game with multiple males playing.” Then, we ask the system to determine if this sentence, which we’ll call the hypothesis H, is true or false: “Some men are playing a sport.” In order to know that the hypothesis is true, the system has to use the fact that “soccer” is a “sport” and that “males” is similar (roughly) to “Some men.”

A premise and hypothesis pair along with a relevant subgraph from ConceptNet. Blue concepts occur in the premise, green in the hypothesis, and purple connect them.

Reading between the lines

Most approaches to textual entailment only use the text in each sentence to learn the task. However, when humans understand natural language, we tend to go beyond the words on the page by putting them into context with what we already know. In essence, humans’ ability to “read between the lines” is key to our understanding of language.

To model this notion, we developed an automated approach for solving this problem that uses symbolic knowledge from external sources, such as knowledge graphs (KG), that could help solve the problem.

For example, in the knowledge graph known as ConceptNet, SOCCER is a type of SPORT. This knowledge is necessary for understanding that “Some men are playing a sport” follows from “A soccer game with multiple males playing.”

KG-augmented Entailment System (KES)

With our proposed KG-augmented Entailment System (KES), we combine a standard language model with a graph convolutional neural network (GCN).

Primary components of KES: standard text-based model, GCN-based graph embedder, and final feedforward classifier.

The former (e.g., BERT), is used as in related works to encode the two sentences. We then use us a GCN to get an embedding for the knowledge graph information, concatenate this to the other embedding, and apply a simple feedforward network for the final decision: entails, contradicts, or is neutral.

The role of the Graph Convolutional Network (GCN)

The application of the GCN in this way makes this approach novel. We first have to extract relevant parts of existing knowledge graphs since they are too large and noisy to be used entirely. To this end, we focus on one knowledge graph (e.g. ConceptNet), extract the 1-hop neighborhood for all concepts that occur in the given sentences, then apply personalized page rank to extract those concepts and relationships that are most relevant in our context. We extend this subgraph by adding two super nodes representing the premise and hypothesis, which are connected to all the nodes in their respective sentences. In this way, we obtain a contextual subgraph of reasonable size, which we encode using a relational GCN.

Finally, we concatenate the embedding of the contextual subgraph with text embeddings to create a final representation for the premise and hypothesis, which is fed into a standard feedforward neural network classifier.

Overview of the KES approach: KES links terms in the premise and hypothesis to concepts in KG, creates contextual subgraphs via personalized page rank filtering, encodes those subgraphs with an R-GCN, and finally combines the aggregated node embeddings with text representations into a feedforward classifier. hp and hh in the figure denote hLvp and hLvh in Equa- tion (6) respectively.

Experimental Results

To evaluate our approach, we use four standard datasets for NLI: Scitail, MultiNLI, SNLI, BreakingNLI.

The addition of graph embeddings improves performance on a number of text-based models. Interestingly, the addition of graph embeddings most strikingly improved performance on the challenging BreakingNLI dataset (which contains an adversarial test set where most of the common text-based approaches show significant drop in performance.).

Table 1: Entailment accuracy results of KES with different text models compared to text-only entailment models. Bold values indicate where KES improves performance.

Summary

Through this research, we demonstrate that structured, symbolic external knowledge can improve the outcome of learning models for solving textual entailment. This is an important area of research which is still largely unexplored but provides a lot of potential for improving learning and making it more interpretable.