爱可可AI前沿推介(7.17)

LG - 机器学习 CV - 计算机视觉 CL - 计算与语言 AS - 音频与语音 RO - 机器人

1、[CV] From Show to Tell: A Survey on Image Captioning

M Stefanini, M Cornia, L Baraldi, S Cascianelli, G Fiameni, R Cucchiara
[University of Modena and Reggio Emilia & NVIDIA AI Technology Centre]
从展示到描述：图像描述生成综述。将视觉和语言联系起来，是生成式智能的重要组成部分。过去几年，在图像描述生成上投入了大量的研究工作，即用句法和语义上有意义的句子来描述图像的任务。从2015年开始，这项任务通常由一个视觉编码步骤和一个文本生成的语言模型组成的管道来解决。这些年，通过对物体区域、属性和关系的利用，以及引入多模态连接、全注意力方法和类似BERT的早期融合策略，两个组件都有了很大的发展。然而，尽管取得了令人印象深刻的成果，图像描述生成的研究还没有达到一个结论性的答案。本文的目的是提供一个全面的概述，并对图像字幕的方法进行分类，从视觉编码和文本生成到训练策略、使用的数据集和评估指标。对许多相关的最先进方法进行了定量比较，以确定图像描述生成架构和训练策略中最有影响的技术创新。此外，还分析和讨论了该问题的许多变体及其公开挑战。本文最终目标是作为一个工具，用于了解现有的最先进的技术，并强调计算机视觉和自然语言处理可以找到最佳协同作用的研究领域的未来方向。
Connecting Vision and Language plays an essential role in Generative Intelligence. For this reason, in the last few years, a large research effort has been devoted to image captioning, i.e. the task of describing images with syntactically and semantically meaningful sentences. Starting from 2015 the task has generally been addressed with pipelines composed of a visual encoding step and a language model for text generation. During these years, both components have evolved considerably through the exploitation of object regions, attributes, and relationships and the introduction of multi-modal connections, fully-attentive approaches, and BERT-like early-fusion strategies. However, regardless of the impressive results obtained, research in image captioning has not reached a conclusive answer yet. This work aims at providing a comprehensive overview and categorization of image captioning approaches, from visual encoding and text generation to training strategies, used datasets, and evaluation metrics. In this respect, we quantitatively compare many relevant state-of-the-art approaches to identify the most impactful technical innovations in image captioning architectures and training strategies. Moreover, many variants of the problem and its open challenges are analyzed and discussed. The final goal of this work is to serve as a tool for understanding the existing state-of-the-art and highlighting the future directions for an area of research where Computer Vision and Natural Language Processing can find an optimal synergy.
https://weibo.com/1402400261/Kp6cdigQw

2、[CL] Tailor: Generating and Perturbing Text with Semantic Controls

A Ross, T Wu, H Peng, M E. Peters, M Gardner
[Allen Institute for Artificial Intelligence & University of Washington]
Tailor：基于语义控制的文本生成和扰动。进行受控的扰动，对于各种任务(例如数据增强)来说是必不可少的，但建立特定任务的生成器可能是昂贵的。本文提出Tailor，一种与任务无关的生成系统，以语义控制的方式扰动文本。通过组合扰动操作，Tailor实现了复杂的和上下文感知的变化，支持各种下游应用。通过非似然训练，设计了Tailor的生成器，以遵循一系列从语义角色衍生出来的控制代码。通过对这些控制代码的修改，Tailor可以产生精细的扰动。本文在控制代码上实现了一套操作，可以组成复杂的扰动策略，并在三个不同的应用中证明其有效性。首先，Tailor有助于构建高质量的对比集，这些对比集在词汇上是多样化的，并且比原始任务测试数据的偏差要小。第二，与自动标签启发式方法配对，Tailor有助于通过数据增强来提高模型的泛化性。通过对训练数据≈5%的扰动，在NLI挑战集上获得1.73的平均增益。第三，在没有任何微调开销的情况下，Tailor的扰动有效地提高了细粒度风格迁移的构成性，在6个迁移上超过了微调基线的表现。
Making controlled perturbations is essential for various tasks (e.g., data augmentation), but building task-specific generators can be expensive. We introduce Tailor, a taskagnostic generation system that perturbs text in a semantically-controlled way. With unlikelihood training, we design Tailor’s generator to follow a series of control codes derived from semantic roles. Through modifications of these control codes, Tailor can produce fine-grained perturbations. We implement a set of operations on control codes that can be composed into complex perturbation strategies, and demonstrate their effectiveness in three distinct applications: First, Tailor facilitates the construction of high-quality contrast sets that are lexically diverse, and less biased than original task test data. Second, paired with automated labeling heuristics, Tailor helps improve model generalization through data augmentation: We obtain an average gain of 1.73 on an NLI challenge set by perturbing just ≈5% of training data. Third, without any finetuning overhead, Tailor’s perturbations effectively improve compositionality in fine-grained style transfer, outperforming fine-tuned baselines on 6 transfers.
https://weibo.com/1402400261/Kp6gNiyot

3、[CL] Wordcraft: a Human-AI Collaborative Editor for Story Writing

A Coenen, L Davis, D Ippolito, E Reif, A Yuan
[Google Research]
Wordcraft：面向故事创作的人-AI协同编辑器。随着神经语言模型效率的提高，它们越来越多地被应用于现实世界中。然而，这些应用在其支持的互动模式方面往往是有限的。本文提出Wordcraft，一种用于故事写作的人工智能辅助编辑器，人类作者和对话系统合作写一个故事。其新颖的界面使用少样本学习和对话的自然能供性来支持各种互动。协同编辑器为作家提供了一个沙盒，以探索基于Transformer的语言模型的界限，并为未来的人在环路训练管道和新的评估方法铺平道路。
As neural language models grow in effectiveness, they are increasingly being applied in real-world settings. However these applications tend to be limited in the modes of interaction they support. In this extended abstract, we propose Wordcraft, an AI-assisted editor for story writing in which a writer and a dialog system collaborate to write a story. Our novel interface uses few-shot learning and the natural affordances of conversation to support a variety of interactions. Our editor provides a sandbox for writers to probe the boundaries of transformer-based language models and paves the way for future human-in-the-loop training pipelines and novel evaluation methods.
https://weibo.com/1402400261/Kp6l990jk

4、[CL] HTLM: Hyper-Text Pre-Training and Prompting of Language Models

A Aghajanyan, D Okhonko, M Lewis, M Joshi, H Xu, G Ghosh, L Zettlemoyer
[Facebook AI]
HTLM：语言模型的超文本预训练和提示。本文提出HTLM，一种在大规模网络爬取中训练的超文本语言模型。建立超文本模型有许多优点。(1)很容易大规模收集，(2)提供了丰富的文档级和与终端任务相关的监督(例如，class和id属性经常编码文档类别信息)，(3)允许遵循HTML既定语义的新的结构化提示(例如，通过为包含输入文本的网页的