Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Evaluation and Prediction #82

Open
mayinghan opened this issue Apr 15, 2020 · 6 comments
Open

Evaluation and Prediction #82

mayinghan opened this issue Apr 15, 2020 · 6 comments

Comments

@mayinghan
Copy link

Hi,

I was trying to run the NER task on a customized dataset. The training process was successful. However, when it went to evaluation and prediction step, the program stuck at INFO:tensorflow:Done running local_init_op. and not moving forward. Is there any potential fix on this problem?

Here is the log

INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Graph was finalized.
2020-04-14 20:00:54.378068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2020-04-14 20:00:54.378117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-14 20:00:54.378127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2020-04-14 20:00:54.378135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2020-04-14 20:00:54.378210: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 23077 MB memory) -> physical GPU (device: 0, name: Quadro P6000, pci bus id: 0000:02:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from output/ner/i2b2/checkpoint/model.ckpt-100
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
@stevezheng23
Copy link
Owner

Hi @mayinghan, I can't repro this issue and might need more information for investigating. For example, full log, config file, data size and environment setting will be helpful

@mayinghan
Copy link
Author

@stevezheng23 thanks for the reply. In the dev set, there are 30426 entities (3209 sentences). In the test set, there are around 442180 entities (45053 sentences).
This is the system environment

NAME="Red Hat Enterprise Linux Workstation"
VERSION="7.7 (Maipo)"

My tensorflow is using 1.13.0 , python version is 3.7.3

@mayinghan
Copy link
Author

This is the log for prediction. The one for evaluation is very similar to this one

INFO:tensorflow:***** Run prediction *****
INFO:tensorflow:  Num examples = 45053
INFO:tensorflow:  Batch size = 8
INFO:tensorflow:Writing example 0 of 45053
INFO:tensorflow:*** Example ***
INFO:tensorflow:guid: 819a3593-164d-49d8-b267-fbe1971305f4
INFO:tensorflow:tokens: ▁RE COR DR ECO RD ▁000 8 ▁ ** IN ST I TU TION
INFO:tensorflow:labels: O X X X X O X O X X X X X X
INFO:tensorflow:input_ids: 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 9322 22787 7398 25133 13049 14180 385 17 4684 5679 6935 96 13775 10829 4 3
INFO:tensorflow:input_masks: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
INFO:tensorflow:segment_ids: 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2
INFO:tensorflow:label_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 2 2 1 2 1 2 2 2 2 2 2 4 3
INFO:tensorflow:*** Example ***
INFO:tensorflow:guid: b723ef52-b650-4db7-9e9b-8757412f53c2
INFO:tensorflow:tokens: ▁G ENE RAL ▁ MED IC INE
INFO:tensorflow:labels: O X X O X X X
INFO:tensorflow:input_ids: 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 457 27791 28742 17 23513 4383 16702 4 3
INFO:tensorflow:input_masks: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0
INFO:tensorflow:segment_ids: 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 0 0 0 0 0 0 0 2
INFO:tensorflow:label_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 1 2 2 2 4 3
INFO:tensorflow:*** Example ***
INFO:tensorflow:guid: 316d992c-16cc-4a94-9919-a0d20bd524b8
INFO:tensorflow:tokens: ▁ ATT END ING ▁ PH Y S IC IAN ▁PRO GR ESS ▁NOTE
INFO:tensorflow:labels: O X X X O X X X X X O X X O
INFO:tensorflow:input_ids: 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 17 21889 12246 5103 17 10668 936 83 4383 26777 13673 13548 15467 13459 4 3
INFO:tensorflow:input_masks: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
INFO:tensorflow:segment_ids: 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2
INFO:tensorflow:label_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 2 1 2 2 2 2 2 1 2 2 1 4 3
INFO:tensorflow:*** Example ***
INFO:tensorflow:guid: 3777c63d-7d55-48d6-85ef-912a206358a1
INFO:tensorflow:tokens: ▁P AT I ENT ▁N AME ▁ :
INFO:tensorflow:labels: O X X X O X O X
INFO:tensorflow:input_ids: 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 395 7813 96 11007 578 23847 17 60 4 3
INFO:tensorflow:input_masks: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0
INFO:tensorflow:segment_ids: 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 0 0 0 0 0 0 0 0 2
INFO:tensorflow:label_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 2 1 2 1 2 4 3
INFO:tensorflow:*** Example ***
INFO:tensorflow:guid: ede694f6-f383-4e3c-ac92-9dc56ea26794
INFO:tensorflow:tokens: ▁ ** N AME [ AA A ▁ , ▁B BB ▁M ]
INFO:tensorflow:labels: O X X X X X X O X O X O X
INFO:tensorflow:input_ids: 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 17 4684 353 23847 10849 4912 246 17 19 322 10124 414 3158 4 3
INFO:tensorflow:input_masks: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
INFO:tensorflow:segment_ids: 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2
INFO:tensorflow:label_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 2 2 2 2 1 2 1 2 1 2 4 3
INFO:tensorflow:Writing example 10000 of 45053
INFO:tensorflow:Writing example 20000 of 45053
INFO:tensorflow:Writing example 30000 of 45053
INFO:tensorflow:Writing example 40000 of 45053
number of examples detected from XLNetInputBuilder 45053
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Running infer on CPU
INFO:tensorflow:*** Features ***
INFO:tensorflow:  name = input_ids, shape = (?, 256)
INFO:tensorflow:  name = input_masks, shape = (?, 256)
INFO:tensorflow:  name = label_ids, shape = (?, 256)
INFO:tensorflow:  name = segment_ids, shape = (?, 256)
INFO:tensorflow:memory input None
INFO:tensorflow:Use float type <dtype: 'float32'>
INFO:tensorflow:Initialize from the ckpt /home/ma.yingha/workspace/py3/ner/xlnet_cased_L-24_H-1024_A-16/xlnet_model.ckpt
INFO:tensorflow:**** Global Variables ****
INFO:tensorflow:  name = model/transformer/r_w_bias:0, shape = (24, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/r_r_bias:0, shape = (24, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/word_embedding/lookup_table:0, shape = (32000, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/r_s_bias:0, shape = (24, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/seg_embed:0, shape = (24, 2, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_0/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_1/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_2/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_3/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_4/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_5/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_6/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_7/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_8/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_9/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_10/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_11/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_12/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_13/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_14/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_15/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_16/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_17/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_18/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_19/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_20/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_21/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_22/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/rel_attn/q/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/rel_attn/k/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/rel_attn/v/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/rel_attn/r/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/rel_attn/o/kernel:0, shape = (1024, 16, 64), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/rel_attn/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/rel_attn/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/ff/layer_1/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/ff/layer_1/bias:0, shape = (4096,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/ff/layer_2/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/ff/layer_2/bias:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/ff/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = model/transformer/layer_23/ff/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*
INFO:tensorflow:  name = ner/dense/kernel:0, shape = (1024, 11)
INFO:tensorflow:  name = ner/dense/bias:0, shape = (11,)
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Graph was finalized.
2020-04-14 20:00:54.378068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2020-04-14 20:00:54.378117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-14 20:00:54.378127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2020-04-14 20:00:54.378135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2020-04-14 20:00:54.378210: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 23077 MB memory) -> physical GPU (device: 0, name: Quadro P6000, pci bus id: 0000:02:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from output/ner/i2b2/checkpoint/model.ckpt-100
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
^CINFO:tensorflow:prediction_loop marked as finished

@stevezheng23
Copy link
Owner

the log looks normal to me, you might want to check the GPU usage to confirm whether the evaluation job is still running, since the evaluation method doesn't print any log while generating the result, and you can also use a smaller evaluation set (10 - 20 examples) to confirm this

@mayinghan
Copy link
Author

@stevezheng23 thanks. Speaking of the cpu usage, I am currently using xlnet large as the pretrain model. My GPU has 24 GiB memory. However, no matter how I decrease batch size and max_seq_length, the model always eats up like 23 GiB. Is that normal?

@stevezheng23
Copy link
Owner

It should not take that much GPU memory for small batch_size and short max_seq_length, but it's possible that most GPU memory is occupied even though not fully utilized.

You can try re-config tf.ConfigProto(allow_soft_placement=True, gpu_options=tf.GPUOptions(per_process_gpu_memory_fraction=xx.xx))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stevezheng23 @mayinghan