ModelZoo Model List

Model Parameters Note
BERT
bert-small-uncased L=6,H=768,A=12
bert-base-uncased L=12,H=768,A=12
bert-large-uncased L=24,H=1024,A=16
alibaba-pai/pai-bert-base-zh L=6,H=768,A=12 Pretrain BERT w/ Chinese datasets
alibaba-pai/pai-bert-small-zh L=4,H=312,A=12
alibaba-pai/pai-bert-tiny-zh L=2,H=128,A=2
DKPLM（知识预训练）
alibaba-pai/pai-dkplm-medical-small-zh L=4,H=768,A=12 Pretrain BERT w/ Medical KG
alibaba-pai/pai-dkplm-medical-base-zh L=12,H=768,A=12
alibaba-pai/pai-dkplm-medical-large-zh 待发布
alibaba-pai/pai-dkplm-small-zh 待发布 Pretrain BERT w/ General KG
alibaba-pai/pai-dkplm-base-zh 待发布
alibaba-pai/pai-dkplm-large-zh 待发布
alibaba-pai/pai-dkplm-1.3b-zh 待发布
alibaba-pai/pai-dkplm-13b-zh 待发布
GEEP（加速版BERT）
alibaba-pai/geep-bert-base-zh
alibaba-pai/geep-bert-large-zh
RoBERTa
hfl/chinese-roberta-wwm-ext L=12,H=768,A=12
hfl/chinese-roberta-wwm-ext-large L=24,H=1024,A=16
roberta-base-en L=12,H=768,A=12
roberta-large-en L=24,H=1024,A=16

cli使用方式

  1. $ easynlp \
  2.    --mode=train \
  3.    --worker_gpu=1 \
  4.    --tables=train.tsv,dev.tsv \
  5.    --input_schema=label:str:1,sid1:str:1,sid2:str:1,sent1:str:1,sent2:str:1 \
  6.    --first_sequence=sent1 \
  7.    --label_name=label \
  8.    --label_enumerate_values=0,1 \
  9.    --checkpoint_dir=./classification_model \
  10.    --epoch_num=1  \
  11.    --sequence_length=128 \
  12.    --app_name=text_classify \
  13.    --user_defined_parameters='pretrain_model_name_or_path=bert-small-uncased'

代码使用方式

args = initialize_easynlp()
train_dataset = ClassificationDataset(xxx)
model = SequenceClassification(pretrained_model_name_or_path='bert-small-uncased')
Trainer(model=model,  train_dataset=train_dataset).train()

预训练技术总览

image.png