训练与预测：文本生成

更新时间：2022-07-28

生成任务位于wenxin_appzoo/tasks/text_generation
模型均存放于wenxin_appzoo/wenxin_appzoo/models_hub文件夹下
- 执行sh download_ernie_3.0_ch.sh下载3.0模型参数，字典与推断所需环境。
- 执行sh download_ernie_3.0_1.5b_ch.sh可下载ERNIE3.0-1.5B模型参数、字典和网络配置文件。
- 执行sh download_ernie_gen_base_ch.sh下载ernie-gen模型参数和字典等

开始训练

进入指定任务的目录，如wenxin_appzoo/wenxin_appzoo/tasks/text_generation

训练ERNIE 3.0

cd ../../models_hub
sh download_ernie_3.0_ch.sh
cd ../tasks/text_generation
# ERNIE3.0-百亿训练
fleetrun --log_dir log ./run_trainer.py  --param_path "./examples/cls_ernie_3.0_dureader_robust_qg_ch.json" 1>log/lanch.log 2>&1
# ERNIE3.0-1.5B训练
fleetrun --log_dir log ./run_trainer.py  --param_path "./examples/cls_ernie_1.5b_dureader_robust_qg_ch.json" 1>log/lanch.log 2>&1

通过上述脚本调用json文件开启训练（注意ERNIE3.0-1.5B单卡训练时也需fleetrun启动）。
训练阶段日志文件于log文件夹下，workerlog.N 保存了第N张卡的log日志内容，如遇到程序报错可以通过查看不同卡的workerlog.N定位到有效的报错信息。
训练模型保存于./output/cls_ernie_3.0_dureader_robust_qg_ch文件夹下。

训练ERNIE-Gen

cd ../../models_hub
sh download_ernie_gen_base_ch.sh
cd ../tasks/text_generation
python run_trainer_ernie_gen.py --param_path cls_ernie_gen_infilling_ch.json

训练模型保存于./output/ernie_gen_base_ch文件夹下

开始预测

ERNIE 3.0预测

ERNIE3.0 checkpoints转化

将训练过程中保存的checkpoints格式参数转成预测推理的inference_model参数格式，配置文件如下：
注：若ERNIE 3.0-1.5B单卡运行且保存了预测模型时，可跳过使用转化工具这个步骤，直接预测即可。

{
  "dataset_reader": {
    "train_reader": {
      "name": "train_reader",
      "type": "ErnieGenerationReader",
      "fields": [
        {
          "name": "text_a",
          "data_type": "string",
          "reader": null,
          "tokenizer": null,
          "need_convert": true,
          "vocab_path": "",
          "max_seq_len": 512,
          "truncation_type": 0,
          "padding_id": 0
        },
        {
          "name": "text_b",
          "data_type": "string",
          "reader": null,
          "tokenizer": null,
          "need_convert": true,
          "vocab_path": "",
          "max_seq_len": 512,
          "truncation_type": 0,
          "padding_id": 0
        }
      ],
      "config": {
        "data_path": "./data/dureader_robust_qg/train",
        "shuffle": true,
        "batch_size": 1,
        "epoch": 10,
        "sampling_rate": 1.0,
        "need_data_distribute": true,
        "extra_params":{
          "vocab_path":"../../models_hub/ernie_3.0_ch_dir/vocab.txt",
          "label_map_config":"",
          "max_seq_len":32,
          "do_lower_case":true,
          "in_tokens":false,
          "tokenizer": "FullTokenizer",
          "is_seq2seq":true,
          "is_unidirectional":true,
          "generation_start_token":"[HD]"
        }
      }
    }
  },
  "model": {
    "type": "Ernie3Generation",
    "embedding": {
      "emb_dim": 4096,
      "use_amp": true,
      "mem_len": 0,
      "enc_output_name": "nlg_",
      "weight_sharing": false,
      "training_server": false,
      "config_path": "../../models_hub/ernie_3.0_ch_dir/ernie_config.json"
    },
    "optimization":{
      "learning_rate": 0,
      "use_lr_decay": true,
      "use_default_decay": false,
      "lr_scheduler": "linear_warmup_decay",
      "epsilon": 1e-6,
      "warmup_steps": 0,
      "warmup_proportion": 0.1,
      "weight_decay": 0.01,
      "use_dynamic_loss_scaling": false,
      "init_loss_scaling": 524288,
      "incr_every_n_steps": 1000,
      "decr_every_n_nan_or_inf": 2,
      "incr_ratio": 2.0,
      "decr_ratio": 0.5,
      "use_layer_decay": false,
      "layer_decay_ratio": 0.95,
      "n_layers": 60,
      "sharing_layers": 48
    }
  },
  "trainer": {
    "type" : "CustomGenerationTrainerErnie3",
    "PADDLE_PLACE_TYPE": "cpu", # 固定为cpu方式保存inference_model
    "PADDLE_IS_FLEET": 1,
    "is_recompute": false,
    "ramdom_seed": 1,
    "use_amp": false,
        "use_sharding": false,
    "save_inference_model": true,
    "use_fast_executor": true,
    "train_log_step": 10,
    "save_final_ckpt": false,
    "is_do_train": 1,
    "is_eval_dev": 0,
    "is_eval_test": 0,
    "eval_step": 50,
    "save_model_step": 10000000,
    "load_parameters": "",
    "load_checkpoint": "",
    "pre_train_model": [
      {
        "name": "ernie_3.0_ch",
        "params_path": "../../models_hub/ernie_3.0_ch_dir/params"
      }
    ],
    "output_path": "./output/cls_ernie_3.0_dureader_robust_qg_ch"
  }
}

设置学习率learning_rate为0，load_parameters中填入训练时保存的checkpoints的文件路径。
运行如下脚本保存inference_model，耗时大约20分钟。

export CPU_NUM='1'
export CUDA_VISIBLE_DEVICES=0 # mask out other gpus for saving inference-model

fleetrun --log_dir log ./run_trainer.py  --param_path "./examples/cls_ernie_3.0_dureader_robust_qg_ch_save_infer_from_ckpt.json" 1>log/lanch.log 2>&1

预测模型保存于./output/cls_ernie_3.0_dureader_robust_qg_ch/save_inference_model路径下

预测配置文件如下(./examples/cls_ernie_3.0_dureader_robust_qg_ch_infer.json)：

{
  "dataset_reader": {
    "predict_reader": {
      "name": "predict_reader",
      "type": "ErnieGenerationReader",
      "fields": [
        {
          "name": "text_a",
          "data_type": "string",
          "reader": null,
          "tokenizer": null,
          "need_convert": true,
          "vocab_path": "",
          "max_seq_len": 428,
          "truncation_type": 0,
          "padding_id": 0
        },
        {
          "name": "text_b",
          "data_type": "string",
          "reader": null,
          "tokenizer": null,
          "need_convert": true,
          "vocab_path": "",
          "max_seq_len": 428,
          "truncation_type": 0,
          "padding_id": 0
        }
      ],
      "config": {
        "data_path": "./data/dureader_robust_qg/dev",
        "shuffle": false,
        "batch_size": 1, 
        "epoch": 1,
        "sampling_rate": 1.0,
        "need_data_distribute": true,
        "extra_params":{
          "vocab_path":"../../models_hub/ernie_3.0_ch_dir/vocab.txt",
          "label_map_config":"",
          "max_seq_len": 428, # 最长文本配置，将对text_a进行截断。注意：请确保max_seq_len + max_dec_len <= 512，否则会导致显存溢出。
          "do_lower_case":true,
          "in_tokens":false,
          "tokenizer": "FullTokenizer",
          "is_seq2seq":true,
          "is_unidirectional":true,
          "generation_start_token":"[HD]",
          "topk": 1,
          "max_dec_len": 64
        }
      }
    }
  },
  "inference": {
    "type": "CustomGenerationInferenceErnie3",
    "output_path": "./output/predict_result.txt",
    "PADDLE_PLACE_TYPE": "gpu",
    "turn_on_trt": true,
    "use_cache": false,
    "num_labels": 2,
    "thread_num": 2,
    "inference_model_path": "./output/cls_ernie_3.0_dureader_robust_qg_ch/save_inference_model/inference_step_1", # inference_model参数路径（选定上一步转化出的inference_model即可）
    "config_path": "../../models_hub/ernie_3.0_ch_dir/ernie_config.json",
    "extra_param": {
      "meta":{
        "job_type": "text_generation"
      },
      "max_seq_len": 512
    }
  }
}

启动ERNIE 3.0预测

ERNIE 3.0百亿模型预测：

# 下面为inference必须要导入的环境变量以及依赖的python
BASE_PATH="../../models_hub/ernie_3.0_ch_dir/infer_env/" # inference所需环境路径
export CUDA_VISIBLE_DEVICES=0 # 屏蔽其它显卡
export PATH="${BASE_PATH}/py37/bin/:$PATH"
export PYTHONPATH="${BASE_PATH}/py37/"
# ERNIE 3.0百亿模型依赖TensorRT以fp16精度进行单卡预测，下面为所需的tensorRT库，以及对应的cuda和cuddn版本
export LD_LIBRARY_PATH=$BASE_PATH:${BASE_PATH}/cuda-11.0.3/lib64:${BASE_PATH}/cudnn-11.0-linux-x64-v8.0.5.39/lib64:${BASE_PATH}/TensorRT-7.2.1.6/lib:/home/opt/nvidia_lib:${BASE_PATH}/libs:$LD_LIBRARY_PATH
export FLAGS_allocator_strategy=auto_growth # for inference，没有这一行会报显存溢出的错误

mkdir -p log
fleetrun --log_dir log ./run_infer_ernie3.py.py  --param_path "./examples/cls_ernie_3.0_dureader_robust_qg_ch_infer.json" 1>log/lanch.log 2>&1

ERNIE 3.0-1.5B模型预测：

python ./run_infer_ernie3.py  --param_path "./examples/cls_ernie_1.5b_dureader_robust_qg_ch_infer.json"

预测结果保存于./output/predict_result.txt文件中。

ERNIE-Gen预测

python run_infer.py --param_path examples/cls_ernie_gen_infilling_ch_infer.json

准备工作：文本生成

阅读理解