What is Self-Supervised Learning? Why is it diferent?
Self-Supervised Representation Learning
Token-Level:W2V, LM, Masked LM.
- CBOW,SkipGram
-
Beyond Token-Level: Sentence, Discourse etc.
Skip-Thought Vector:基于当前句子预测前向上一个句子和后向的后一个句子
- NextSentence Predict:BERT
- Predict Right Order:Albert
Task-Based Self-Supervised Learning
Examples of Task-Based SSL in NLP
Self-Supervised Learning for Contextualized Extractive Summarization(Wang et al., ACL 2019)
抽取式摘要挖句子,根据原文预测挖空句
Pretrained Encyclopedia: WKLM(Xiong et al.,2020)
预测wiki数据里的entity是否正确
Knowledge-Grounded Self-Supervised Generation.
某些任务需要充分的知识信息,于是在pretrain的时候引入了外部知识信息
KGPT:先把wiki数据转graph,利用wiki的链接构造图,建立了Graph2Text的data
拿着图去找句子,找那些实体和词汇重复较高的较高数据的句子
(大致,这里没有理解)
Self-Supervised Storytelling via Adversarial Learning.
对抗学习,一个发掘这个是不是机器写的,一个和人的标注BLEU
这类方法对无法清晰定义reward的方法使用,因为用了对抗方法来优化reward的计算