- 分享主题：MoE
- 论文标题：Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts
- 论文链接：https://dl.acm.org/doi/pdf/10.1145/3219819.3220007
- 分享人：唐共勇

1. Summary

【必写】，推荐使用 grammarly 检查语法问题，尽量参考论文 introduction 的写作方式。需要写出

  1. 这篇文章解决了什么问题？
  2. 作者使用了什么方法（不用太细节）来解决了这个问题？
  3. 你觉得你需要继续去研究哪些概念才会加深你对这篇文章的理解？

This paper proposes a novel multi-task learning method, namely the Multi Gate expert combination (MMOE) model. MMOE explicitly models task relationships and learns task-specific functions to take advantage of shared representation. It allows automatic assignment of parameters to capture shared or task-specific information, avoiding the need to add many new parameters to each task. By sharing the expert sub-model in all tasks, the structure of a mixture of experts (MOE) is adapted to multi-task learning. At the same time, a gating network is trained to optimize each task. Gating networks use input features and output softmax gates to assemble experts with different weights, thus allowing different tasks to use experts in different ways. Then the results of the experts are transferred to the tower network of specific tasks. In this way, the gating network of different tasks can learn the mixed pattern of different expert aggregation, to capture the task relationship.

2. 你对于论文的思考

需要写出你自己对于论文的思考，例如优缺点，你的takeaways

  1. MOE与之前hard参数的共享机制不同，这里加入了gate门控系统。在hard共享机制中，无论输入的样本特征如何变化，共享机制不变。MOE各任务之间会根据输入特征的不同来组合选取不同的expert（后面也会称之为专家网络，这里的专家网络可以理解为具体的一个网络模块）输出。

MMOE，在MOE的基础上升级，为每一个任务建立了一个单独的gate门控系统。

  1. 门控神经网络来整合信息，对于我的多任务模型来说，可以借助聚类门控来整合信息

    3. 其他

    【可选】

image.png
专家组合
原始专家组合 (MoE) 模型 [21] 可以表述为：
Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts - 图2
其中， Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts - 图3 ，同时， Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts - 图4 是 Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts - 图5 输出的第 Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts - 图6 个logits对数，表示专家 Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts - 图7 的概率。
这里， Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts - 图8 是 Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts - 图9 个专家网络，同时 Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts - 图10 代表一个门控网络集合了所有专家的结果。 更具体地说，门控网络 Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts - 图11 根据输入生成 Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts - 图12 个专家的分布，最终输出是所有专家输出的加权和。