文心工具日志说明
更新时间：2021-03-23
默认日志说明
wenxin各入口脚本运行的日志会自动保存在对应任务目录下的./log/test.log中。 以训练文本分类任务cls_cnn_ch.json为例，生成的部分日志及说明如下所示：
INFO: 02-24 08:00:35: base_dataset_reader.py:96 * 139822175426304 set data_generator and start.......
INFO: 02-24 08:00:35: custom_trainer.py:64 * 139822175426304 epoch 2 progress 176/199 pyreader queue size 50 ## epoch：当前训练轮次；progress：当前训练进度；pyreader queue size：当前pyreader队列大小
DEBUG: 02-24 08:00:35: cnn_classification.py:148 * 139822175426304 phase = training acc = 0.625 precision = 0.3125 step = 20 time_cost = 0.499283075333 ## phase：该log类型，train为训练log，test为测试log，dev为评估log；acc、precision等：评估指标；step：当前训练步数；time_cost：当前batch所消耗的时间
...
INFO: 02-24 08:00:38: base_dataset_reader.py:96 * 139822175426304 set data_generator and start.......
DEBUG: 02-24 08:00:38: cnn_classification.py:152 * 139822175426304 phase = test acc = 0.54 precision = 0.27 time_cost = 0.105578899384
...
INFO: 02-24 08:00:41: custom_trainer.py:64 * 139822175426304 epoch 9 progress 198/199 pyreader queue size 9
DEBUG: 02-24 08:00:41: cnn_classification.py:148 * 139822175426304 phase = training acc = 0.75 precision = 0.375 step = 240 time_cost = 0.474550008774
INFO: 02-24 08:00:42: custom_trainer.py:104 * 139822175426304 Final test result: ## 此次训练结束时的最终评估
INFO: 02-24 08:00:42: base_dataset_reader.py:96 * 139822175426304 set data_generator and start.......
DEBUG: 02-24 08:00:42: cnn_classification.py:152 * 139822175426304 phase = test acc = 0.53 precision = 0.267676767677 time_cost = 0.0935051441193
INFO: 02-24 08:00:42: run_with_json.py:96 * 139822175426304 end of run train and eval .....
INFO: 02-24 08:00:42: run_with_json.py:98 * 139822175426304 os exit.自定义日志
如果您需要根据自己的需求进行日志的定义，请在日志配置在./wenxin/utils/log.py中，可根据需要修改相关配置。init_log()及其参数说明如下所示：
def init_log(log_path, level=logging.INFO, when="D", backup=7,
format="%(levelname)s: %(asctime)s: %(filename)s:%(lineno)d * %(thread)d %(message)s",
datefmt="%m-%d %H:%M:%S"):
"""
init_log - initialize log module
Args:
log_path - Log file path prefix.
Log data will go to two files: log_path.log and log_path.log.wf
Any non-exist parent directories will be created automatically
level - msg above the level will be displayed
DEBUG < INFO < WARNING < ERROR < CRITICAL
the default value is logging.INFO
when - how to split the log file by time interval
'S' : Seconds
'M' : Minutes
'H' : Hours
'D' : Days
'W' : Week day
default value: 'D'
format - format of the log
default format:
%(levelname)s: %(asctime)s: %(filename)s:%(lineno)d * %(thread)d %(message)s
INFO: 12-09 18:02:42: log.py:40 * 139814749787872 HELLO WORLD
backup - how many backup file to keep
default value: 7
Raises:
OSError: fail to create log directories
IOError: fail to open log file
"""
formatter = logging.Formatter(format, datefmt)
logger = logging.getLogger()
logger.setLevel(level)
dir = os.path.dirname(log_path)
if not os.path.isdir(dir):
os.makedirs(dir)
handler = logging.handlers.TimedRotatingFileHandler(log_path + ".log",
when=when,
backupCount=backup)
handler.setLevel(level)
handler.setFormatter(formatter)
logger.addHandler(handler)
handler = logging.handlers.TimedRotatingFileHandler(log_path + ".log.wf",
when=when,
backupCount=backup)
handler.setLevel(logging.WARNING)
handler.setFormatter(formatter)
logger.addHandler(handler)
console_hdl = logging.StreamHandler()
console_hdl.setFormatter(formatter)
logger.addHandler(console_hdl)