Grammar-Based Grounded Lexicon Learning
Authors
Authors
- Roger Levy
- Joshua Tenenbaum
- Jiayuan Mao
- Freda H. Shi
- Jiajun Wu
Authors
- Roger Levy
- Joshua Tenenbaum
- Jiayuan Mao
- Freda H. Shi
- Jiajun Wu
Published on
12/14/2021
Categories
We present Grammar-Based Grounded Lexicon Learning (G2L2), a lexicalist approach toward learning a compositional and grounded meaning representation of language from grounded data, such as paired images and texts. At the core of G2L2 is a collection of lexicon entries, which map each word to a tuple of a syntactic type and a neuro-symbolic semantic program. For example, the word shiny has a syntactic type of adjective; its neuro-symbolic semantic program has the symbolic form λx.filter(x, SHINY), where the concept SHINY is associated with a neural network embedding, which will be used to classify shiny objects. Given an input sentence, G2L2 first looks up the lexicon entries associated with each token. It then derives the meaning of the sentence as an executable neuro-symbolic program by composing lexical meanings based on syntax. The recovered meaning programs can be executed on grounded inputs. To facilitate learning in an exponentiallygrowing compositional space, we introduce a joint parsing and expected execution algorithm, which does local marginalization over derivations to reduce the training time. We evaluate G2L2 on two domains: visual reasoning and language-driven navigation. Results show that G2L2 can generalize from small amounts of data to novel compositions of words.
Please cite our work using the BibTeX below.
@inproceedings{
mao2021grammarbased,
title={Grammar-Based Grounded Lexicon Learning},
author={Jiayuan Mao and Freda H. Shi and Jiajun Wu and Roger P. Levy and Joshua B. Tenenbaum},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=iI6nkEZkOl}
}