爱可可AI前沿推介(1.5)

LG - 机器学习 CV - 计算机视觉 CL - 计算与语言 AS - 音频与语音 RO - 机器人 (*表示值得重点关注)

1、[CL] Shortformer: Better Language Modeling using Shorter Inputs

O Press, N A. Smith, M Lewis
[University of Washington & Facebook AI Research & Allen Institute for AI]
Shortformer：用更短的输入实现更好的语言建模。研究了缩短transformer输入长度可能带来的好处，通过先对较短子序列进行训练，再用分阶段训练逐渐过渡到较长子序列，可改善复杂度，减少训练时间。定义了一种新方法，即位置填充注意力(position-infilled attention)，能缓存和有效关注之前计算的表示。使用这种方法的模型，不需要大的输入子序列。两种方法结合，可以产生更快、更准确的语言模型。在不添加任何参数的情况下，提高了65%的训练速度，生成速度提高了9倍，有效提高了WikiText-103上的困惑度。 We explore the benefits of decreasing the input length of transformers. First, we show that initially training the model on short subsequences, before moving on to longer ones, both reduces overall training time and, surprisingly, gives a large improvement in perplexity. We then show how to improve the efficiency of recurrence methods in transformers, which let models condition on previously processed tokens (when generating sequences that are larger than the maximal length that the transformer can handle at once). Existing methods require computationally expensive relative position embeddings; we introduce a simple alternative of adding absolute position embeddings to queries and keys instead of to word embeddings, which efficiently produces superior results. By combining these techniques, we increase training speed by 65%, make generation nine times faster, and substantially improve perplexity on WikiText-103, without adding any parameters. https://weibo.com/1402400261/JBI2AuK8t

2、 [LG] Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimization

C Huang, R T. Q. Chen, C Tsirigotis, A Courville
[University of Montreal & University of Toronto]
凸势流：基于最优传输和凸优化的通用概率分布。提出一种新的使用凸势梯度图的归一化流参数化Convex Potential Flows (CP-Flow)，并将其与最优传输理论联系起来，证明所提出的流是通用密度模型，利用凸优化工具来实现高效的训练和模型反演。通过实验表明了所提出方法的有效性，证明了可通过设计更好的ICNN架构来提高性能。 Flow-based models are powerful tools for designing probabilistic models with tractable density. This paper introduces Convex Potential Flows (CP-Flow), a natural and efficient parameterization of invertible models inspired by the optimal transport (OT) theory. CP-Flows are the gradient map of a strongly convex neural potential function. The convexity implies invertibility and allows us to resort to convex optimization to solve the convex conjugate for efficient inversion. To enable maximum likelihood training, we derive a new gradient estimator of the log-determinant of the Jacobian, which involves solving an inverse-Hessian vector product using the conjugate gradient method. The gradient estimator has constant-memory cost, and can be made effectively unbiased by reducing the error tolerance level of the convex optimization routine. Theoretically, we prove that CP-Flows are universal density approximators and are optimal in the OT sense. Our empirical results show that CP-Flow performs competitively on standard benchmarks of density estimation and variational inference. https://weibo.com/1402400261/JBIblEPZo

3、[LG] Reinforcement Learning for Control of Valves

R Siraskar
[Coventry University]
强化学习阀门控制。利用统一框架对强化学习(RL)与PID(比例-积分-微分)策略在非线性阀门控制上进行了比较。用DDPG(Deep Deterministic Policy-Gradient)算法和Simulink仿真非线性阀门进行训练，结果表明，强化学习控制器在跟踪信号与速度方面表现极好，相对于参考信号产生的误差较小，而PID在干扰抑制方面表现较好，也有利于延长阀门寿命。引入”分级学习(Graded Learning)”，作为对更正式和更为算法化的”强化学习课程(Curriculum for Reinforcement Learning)”的简化和面向应用的变体。实验表明，这有助于收敛复杂非线性现实世界系统的学习任务。 This paper compares reinforcement learning (RL) with PID (proportional-integral-derivative) strategy for control of nonlinear valves using a unified framework. RL is an autonomous learning mechanism that learns by interacting with its environment. It is gaining increasing attention in the world of control systems as a means of building optimal-controllers for challenging dynamic and nonlinear processes. Published RL research often uses open-source tools (Python and OpenAI Gym environments) which could be difficult to adapt and apply by practicing industrial engineers, we therefore used MathWorks tools. MATLAB’s recently launched (R2019a) Reinforcement Learning Toolbox was used to develop the valve controller; trained using the DDPG (Deep Deterministic Policy-Gradient) algorithm and Simulink to simulate the nonlinear valve and setup the experimental test-bench to evaluate the RL and PID controllers. Results indicate that the RL controller is extremely good at tracking the signal with speed and produces a lower error with respect to the reference signals. The PID, however, is better at disturbance rejection and hence provides a longer life for the valves. Experiential learnings gained from this research are corroborated against published research. It is known that successful machine learning involves tuning many hyperparameters and significant investment of time and efforts. We introduce Graded Learning" as a simplified, application oriented adaptation of the more formal and algorithmicCurriculum for Reinforcement Learning’’. It is shown via experiments that it helps converge the learning task of complex non-linear real world systems. https://weibo.com/1402400261/JBIfBv9XQ

4、[CL] kōan: A Corrected CBOW Implementation

O İrsoy, A Benton, K Stratos
[Bloomberg LP & Rutgers University]
纠正(词向量)CBOW实现。NLP社区普遍认为，连续词袋(CBOW)词嵌入往往表现低于跳词(SG)嵌入。本文发现，这种观点并不是建立在其训练目标的理论差异上，而是建立在标准软件库中错误的CBOW实现上，如官方实现word2vec.c和Gensim。基于CBOW的正确实现产生的词嵌入，在各种内部和外部任务上完全达到与SG竞争的性能，同时训练速度是SG的三倍以上。 It is a common belief in the NLP community that continuous bag-of-words (CBOW) word embeddings tend to underperform skip-gram (SG) embeddings. We find that this belief is founded less on theoretical differences in their training objectives but more on faulty CBOW implementations in standard software libraries such as the official implementation word2vec.c and Gensim. We show that our correct implementation of CBOW yields word embeddings that are fully competitive with SG on various intrinsic and extrinsic tasks while being more than three times as fast to train. We release our implementation, kōan, at > this https URL. https://weibo.com/1402400261/JBIm2bKIX

5、[RO] ROS for Human-Robot Interaction

Y Mohamed, S Lemaignan
[University of the West of England]
面向人-机器人交互的机器人操作系统。介绍了一套用于人-机器人交互(HRI)场景的约定和标准接口，旨在与机器人操作系统(ROS)一起使用。它的直接目的是促进许多HRI相关软件工具之间核心功能的互操作性和可重用性，从骨架跟踪、人脸识别到自然语言处理。这些接口被设计成与广泛的HRI应用相关，从高层次的人群模拟，到群体级的社会互动建模，再到具体的人体运动学建模。 Integrating real-time, complex social signal processing into robotic systems — especially in real-world, multi-party interaction situations — is a challenge faced by many in the Human-Robot Interaction (HRI) community. The difficulty is compounded by the lack of any standard model for human representation that would facilitate the development and interoperability of social perception components and pipelines. We introduce in this paper a set of conventions and standard interfaces for HRI scenarios, designed to be used with the Robot Operating System (ROS). It directly aims at promoting interoperability and re-usability of core functionality between the many HRI-related software tools, from skeleton tracking, to face recognition, to natural language processing. Importantly, these interfaces are designed to be relevant to a broad range of HRI applications, from high-level crowd simulation, to group-level social interaction modelling, to detailed modelling of human kinematics. We demonstrate these interface by providing a reference pipeline implementation, packaged to be easily downloaded and evaluated by the community. https://weibo.com/1402400261/JBIpJ8PQT

另外几篇值得关注的论文：

[CL] HateCheck: Functional Tests for Hate Speech Detection Models

HateCheck：仇恨言论检测模型功能测试
P Röttger, B Vidgen, D Nguyen, Z Waseem, H Margetts, J Pierrehumbert
[University of Oxford & The Alan Turing Institute & Utrecht University & University of Sheffield]
https://weibo.com/1402400261/JBIsxwoW9

[LG] Self-Supervised Hyperboloid Representations from Logical Queries over Knowledge Graphs

知识图谱逻辑查询的自监督双曲表示
N Choudhary, N Rao, S Katariya, K Subbian, C K. Reddy
[Virginia Tech & Amazon]
https://weibo.com/1402400261/JBIuhlxzz

[CL] Foundations for Near-Term Quantum Natural Language Processing

近程量子自然语言处理基础
B Coecke, G d Felice, K Meichanetzidis, A Toumi
[Oxford University]
https://weibo.com/1402400261/JBIyM1JWs