2 |
Mapping Natural Language Instructions to Mobile UI Action Sequences ...
|
|
|
|
Abstract:
We present a new problem: grounding natural language instructions to mobile user interface actions, and create three new datasets for it. For full task evaluation, we create PIXELHELP, a corpus that pairs English instructions with actions performed by people on a mobile UI emulator. To scale training, we decouple the language and action data by (a) annotating action phrase spans in HowTo instructions and (b) synthesizing grounded descriptions of actions for mobile user interfaces. We use a Transformer to extract action phrase tuples from long-range natural language instructions. A grounding Transformer then contextually represents UI objects using both their content and screen position and connects them to object descriptions. Given a starting screen and instruction, our model achieves 70.59% accuracy on predicting complete ground-truth action sequences in PIXELHELP. ... : Annual Conference of the Association for Computational Linguistics (ACL 2020) ...
|
|
Keyword:
Computation and Language cs.CL; FOS Computer and information sciences; Machine Learning cs.LG
|
|
URL: https://dx.doi.org/10.48550/arxiv.2005.03776 https://arxiv.org/abs/2005.03776
|
|
BASE
|
|
Hide details
|
|
4 |
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification ...
|
|
|
|
BASE
|
|
Show details
|
|
6 |
Aspect-augmented Adversarial Networks for Domain Adaptation
|
|
|
|
In: MIT Press (2019)
|
|
BASE
|
|
Show details
|
|
7 |
Neuropathology of RAN translation proteins in fragile X-associated tremor/ataxia syndrome
|
|
|
|
BASE
|
|
Show details
|
|
8 |
Neuropathology of RAN translation proteins in fragile X-associated tremor/ataxia syndrome
|
|
|
|
BASE
|
|
Show details
|
|
9 |
The Fabric of Entropy: A Discussion on the Meaning of Fractional Information
|
|
|
|
BASE
|
|
Show details
|
|
10 |
A Fast, Compact, Accurate Model for Language Identification of Codemixed Text ...
|
|
|
|
BASE
|
|
Show details
|
|
11 |
Anticipating Correlative Thinking: A Comparative Analysis of the Laozi and Phaedrus
|
|
Zhang, Yuan. - : University of Alberta. Department of East Asian Studies., 2018
|
|
BASE
|
|
Show details
|
|
12 |
Transfer learning for low-resource natural language analysis
|
|
|
|
BASE
|
|
Show details
|
|
13 |
Ten pairs to tag - Multilingual POS tagging via coarse mapping between embeddings
|
|
|
|
In: MIT Web Domain (2016)
|
|
BASE
|
|
Show details
|
|
14 |
High-order low-rank tensors for semantic role labeling
|
|
|
|
In: MIT Web Domain (2015)
|
|
BASE
|
|
Show details
|
|
15 |
Hierarchical Low-Rank Tensors for Multilingual Transfer Parsing
|
|
|
|
In: MIT Web Domain (2015)
|
|
BASE
|
|
Show details
|
|
16 |
Randomized greedy inference for joint segmentation, POS tagging and dependency parsing
|
|
|
|
In: MIT Web Domain (2015)
|
|
BASE
|
|
Show details
|
|
17 |
Antonymous Adjectives in Disyllabic Lexical Compounds in Mandarin: A Cognitive Linguistics Perspective
|
|
|
|
BASE
|
|
Show details
|
|
18 |
Low-Rank Tensors for Scoring Dependency Structures
|
|
|
|
In: MIT web domain (2014)
|
|
BASE
|
|
Show details
|
|
19 |
Steps to Excellence: Simple Inference with Refined Scoring of Dependency Trees
|
|
|
|
In: MIT web domain (2014)
|
|
BASE
|
|
Show details
|
|
20 |
Spatial Representation of Topological Concepts IN and ON: A Comparative Study of English and Mandarin Chinese
|
|
|
|
BASE
|
|
Show details
|
|
|
|