One 、 Reference material

TensorFlow Network model migration and training

Two 、 Related introduction

2.1 Problem introduction

NPU yes AI The development trend of computing power , But at present, most training and online reasoning scripts are still based on GPU. because NPU And GPU The difference in Architecture , be based on GPU Training and online reasoning scripts cannot be directly in NPU Upper use , Need to convert to support NPU Before using .

Rise 910 AI The processor is Huawei 2019 Artificial intelligence released in （AI） Dedicated neural network processor , Its computational power is as high as 256T, The latest calculators are up to 310T, It is the computing power of mainstream processors in the industry 2 times , It can be matched with MindSpore Training framework . At present, most training scripts in the industry are based on TensorFlow Of Python API Development , Run on by default CPU/GPU/TPU On . In order to enable it to take advantage of shengteng 910 AI The surging computing power of the processor performs training , Improve training performance , We need to do relevant migration and adaptation work for the training network script .

2.2 Tensorflow GPU2Ascend

Ascend The platform provides Tensorflow 1.15 Network migration tools , The tool will be based on TensorFlow Of Python API The developed training script is migrated to shengteng AI Perform training on the processor , And achieve the best performance of training accuracy . This tool is suitable for native Tensorflow Training script migration scenario ,AI Algorithm engineers use this tool to analyze native TensorFlow Python API and Horovod Python API It's rising AI Support on the processor , At the same time, the original TensorFlow The training script is automatically migrated to shengteng AI Scripts supported by the processor . For those that cannot be migrated automatically API, You can refer to the migration report output by the tool , Adapt and modify the training script accordingly .

be based on TensorFlow Of Python API The developed training script is migrated to shengteng AI Perform training on the processor , At present, there are two ways of migration ： Automatic migration mode and Manual migration . among , Manual migration is more complicated , It is recommended to give priority to automatic migration .

3、 ... and 、 Important note

1. It's rising AI Before the processor migrates the model , Users are advised to prepare in advance based on **TensorFlow 1.15 ** Developed training model and supporting data set , and Ask for in GPU or CPU Run through , Accuracy convergence , And meet the expected accuracy and performance requirements . At the same time, record relevant Accuracy and performance index , Used for subsequent rising AI Compare the accuracy and performance of the processor .
2. Dynamic is not supported shape Network migration .
3. I won't support it tf.keras And native Keras Of Loss Scale Function migration .

Four 、 preparation

1. Download the source code ;

   With tensorflow Version of yolov3 For example , Download the source code tensorflow-yolov3;

   Blogger's warehouse ：https://gitee.com/lljyoyo1995/tensorflow-yolov3

   #  Scripts to be migrated 
   https://gitee.com/lljyoyo1995/tensorflow-yolov3.git
   
   #  Script after migration 
   https://gitee.com/lljyoyo1995/tensorflow-yolov3_npu.git
   

2. Get ready VOC Data sets ;

   Download datasets

   $ wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
   $ wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
   $ wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
   

   Decompress the dataset , Organize data sets

   VOC           # VOC The path where the dataset is located 
   ├── test
   |    └──VOCdevkit
   |        └──VOC2007 (from VOCtest_06-Nov-2007.tar)
   └── train
        └──VOCdevkit
            └──VOC2007 (from VOCtrainval_06-Nov-2007.tar)
            └──VOC2012 (from VOCtrainval_11-May-2012.tar)
   

   Generate txt Mark the file

   Generate voc_train.txt and voc_test.txt Mark the file .

   python scripts/voc_annotation.py --data_path /PATH/TO/VOC
   

3. stay CPU perhaps GPU Run through the code ;

4. Upload the code to shengteng server , Bloggers upload to ModelArts Training on the platform ;

5、 ... and 、 Key steps

5.1 Running environment

5.2 tfplugin Migration tools

/home/ma-user/Ascend/tfplugin/latest/python/site-packages/npu_bridge/convert_tf2npu

[[email protected] convert_tf2npu]$ ll
total 372
drwxr-x--- 2 ma-user ma-group   4096 Mar  4 18:20 __pycache__
-rw-rw-r-- 1 ma-user ma-group  37897 Mar  4 18:20 ast_impl.py
-rw-rw-r-- 1 ma-user ma-group   2954 Mar  4 18:20 conver.py
-rw-rw-r-- 1 ma-user ma-group   5020 Mar  4 18:20 conver_by_ast.py
-rw-rw-r-- 1 ma-user ma-group   9965 Mar  4 18:20 file_op.py
-rw-rw-r-- 1 ma-user ma-group   5925 Mar  4 18:20 main.py  # linux Environmental Science 
-rw-rw-r-- 1 ma-user ma-group   8769 Mar  4 18:20 main_win.py  # windows Environmental Science 
drwxr-x--- 2 ma-user ma-group   4096 Mar  4 18:20 mappings
-rw-rw-r-- 1 ma-user ma-group 255453 Mar  4 18:20 tf1.15_api_support_list.xlsx
-rw-rw-r-- 1 ma-user ma-group   4619 Mar  4 18:20 tf_func_def.py
-rw-rw-r-- 1 ma-user ma-group   6203 Mar  4 18:20 util.py
-rw-rw-r-- 1 ma-user ma-group   1299 Mar  4 18:20 util_global.py
-rw-rw-r-- 1 ma-user ma-group   5870 Mar  4 18:20 visit_by_ast.py

Operation instruction

cd /home/ma-user/Ascend/tfplugin/latest/python/site-packages/npu_bridge/convert_tf2npu
python3 main.py -i /root/models/official/resnet

5.3 Training script scanning and automatic migration

cd /usr/local/Ascend/tfplugin/latest/python/site-packages/npu_bridge/convert_tf2npu

python main.py -i /PATH/TO/tensorflow-yolov3 -o /PATH/TO -r /PATH/TO

[ma-user convert_tf2npu]$python main.py -i /home/ma-user/work/MyDocuments/tensorflow-yolov3 -o /home/ma-user/work/MyDocuments -r /home/ma-user/work/MyDocuments
Begin conver, input file: /home/ma-user/work/MyDocuments/tensorflow-yolov3

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/image_demo.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/image_demo.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/from_darknet_weights_to_ckpt.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/from_darknet_weights_to_ckpt.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/LICENSE
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/evaluate.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/evaluate.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/video_demo.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/video_demo.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/convert_weight.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/convert_weight.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/freeze_graph.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/freeze_graph.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/README.md
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/LICENSE.fuck
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.gitignore
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/from_darknet_weights_to_pb.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/from_darknet_weights_to_pb.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/train.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/train.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/yolov3_coco.pb
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/checkpoint/yolov3_coco_demo.ckpt.index
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/checkpoint/checkpoint
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/checkpoint/yolov3_coco.ckpt.index
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/checkpoint/yolov3_coco_demo.ckpt.meta
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/checkpoint/yolov3_coco_demo.ckpt.data-00000-of-00001
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/checkpoint/yolov3_coco.ckpt.meta
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/checkpoint/.DS_Store
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/checkpoint/yolov3_coco.ckpt.data-00000-of-00001
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/main.py
No Tensorflow module is imported in script main.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/main.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/__init__.py
No Tensorflow module is imported in script __init__.py.
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/remove_space.py
No Tensorflow module is imported in script remove_space.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/remove_space.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/remove_delimiter_char.py
No Tensorflow module is imported in script remove_delimiter_char.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/remove_delimiter_char.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/find_class.py
No Tensorflow module is imported in script find_class.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/find_class.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/remove_class.py
No Tensorflow module is imported in script remove_class.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/remove_class.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/result.txt
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/convert_pred_yolo.py
No Tensorflow module is imported in script convert_pred_yolo.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/convert_pred_yolo.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/intersect-gt-and-pred.py
No Tensorflow module is imported in script intersect-gt-and-pred.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/intersect-gt-and-pred.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/convert_keras-yolo3.py
No Tensorflow module is imported in script convert_keras-yolo3.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/convert_keras-yolo3.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/README.md
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/class_list.txt
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/convert_gt_xml.py
No Tensorflow module is imported in script convert_gt_xml.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/convert_gt_xml.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/rename_class.py
No Tensorflow module is imported in script rename_class.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/rename_class.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/convert_pred_darkflow_json.py
No Tensorflow module is imported in script convert_pred_darkflow_json.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/convert_pred_darkflow_json.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/convert_gt_yolo.py
No Tensorflow module is imported in script convert_gt_yolo.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/mAP/extra/convert_gt_yolo.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/scripts/show_bboxes.py
No Tensorflow module is imported in script show_bboxes.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/scripts/show_bboxes.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/scripts/voc_annotation.py
No Tensorflow module is imported in script voc_annotation.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/scripts/voc_annotation.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/data/dataset/voc_train.txt
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/data/dataset/voc_test.txt
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/data/classes/coco.names
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/data/classes/voc.names
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/data/log/events.out.tfevents.1657273471.LAPTOP-4DTD5D42
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/data/anchors/basline_anchors.txt
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/data/anchors/coco_anchors.txt
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/index
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/description
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/packed-refs
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/HEAD
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/config
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/refs/heads/master
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/refs/remotes/origin/HEAD
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/info/exclude
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/pre-receive.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/pre-push.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/commit-msg.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/pre-rebase.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/pre-commit.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/post-update.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/pre-merge-commit.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/push-to-checkout.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/pre-applypatch.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/update.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/prepare-commit-msg.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/fsmonitor-watchman.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/hooks/applypatch-msg.sample
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/objects/pack/pack-e9fd50bffc7fd46014db51ca22d5e478d1f7c993.pack
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/objects/pack/pack-e9fd50bffc7fd46014db51ca22d5e478d1f7c993.idx
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/logs/HEAD
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/logs/refs/heads/master
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.git/logs/refs/remotes/origin/HEAD
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.idea/tensorflow-yolov3.iml
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.idea/workspace.xml
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.idea/.gitignore
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.idea/misc.xml
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.idea/modules.xml
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.idea/vcs.xml
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/docs/requirements.txt
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/docs/Box-Clustering.ipynb
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/docs/images/.jpg
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/docs/images/road.mp4
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/docs/images/road.jpeg
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/docs/images/611_result.jpg
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/.github/FUNDING.yml
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/dataset.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/dataset.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/common.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/common.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/yolov3.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/yolov3.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/__init__.py
No Tensorflow module is imported in script __init__.py.
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/config.py
No Tensorflow module is imported in script config.py.
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/config.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/utils.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/utils.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/backbone.py
Finish conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/backbone.py

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/__pycache__/common.cpython-36.pyc
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/__pycache__/__init__.cpython-36.pyc
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/__pycache__/yolov3.cpython-36.pyc
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/__pycache__/dataset.cpython-36.pyc
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/__pycache__/utils.cpython-36.pyc
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/__pycache__/config.cpython-36.pyc
Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/core/__pycache__/backbone.cpython-36.pyc
1.In brief: Total API: 200, in which Support: 187, API support after migration: 4, Network training support after migration: 9, Not support but no impact on migration: 0, Not support or recommended: 0, Compatible: 0, Deprecated: 0, Analysing: 0
2.After eliminate duplicate: Total API: 63, in which Support: 60, API support after migration: 2, Network training support after migration: 1, Not support but no impact on migration: 0, Not support or recommended: 0, Compatible: 0, Deprecated: 0, Analysing: 0
Finish conver, output file: /home/ma-user/work/MyDocuments; report file: /home/ma-user/work/MyDocuments/report_npu_20220712170053

5.4 Migration completed

tensorflow-yolov3  #  The original script path to be migrated 
report_npu_20220712164226  #  The path of the generated migration report 
tensorflow-yolov3_npu_20220712164226  #  Script path after migration 

5.5 Training models

Set the environment variable

#  Shengteng device installation and deployment software package Ascend-cann-nnae
source /usr/local/Ascend/nnae/set_env.sh

# tfplugin Packet dependency 
source /usr/local/Ascend/tfplugin/set_env.sh

Pull up the training process

python train.py

NPU Resource usage

Memory footprint

6、 ... and 、FAQ

Q： Migration error

Begin conver file: /home/ma-user/work/MyDocuments/tensorflow-yolov3/from_darknet_weights_to_ckpt                                           .py
SyntaxError('invalid syntax', ('<unknown>', 5, 45, "darknet_weights = '<your yolov3.weights' pat h>'\n"))
ERROR:There is a format problem in the script, please check the python code specification or whe                                           ther it is converted to a linux file through 'dos2unix'

 terms of settlement ：
 Modify the wrong code , Re migrate 
 Other similar problems , It can also be solved according to this idea 

darknet_weights = '<your yolov3.weights' path>'
 Change it to 
darknet_weights = '<your yolov3.weights path>'

Q：The input shape of GeOp5_0 is dynamic…

Huawei ModelArts Training Alexnet Model

tensorflow.python.framework.errors_impl.InternalError: The input shape of GeOp5_0 is dynamic, please ensure that npu option[dynamic_input] is set correctly, for more details please refer to the migration guide.
	 [[{
    {
    node GeOp5_0}}]]

 terms of settlement ：
 After the migration train.py Add 

config = tf.ConfigProto(allow_soft_placement=True)
custom_op = config.graph_options.rewrite_options.custom_optimizers.add()
custom_op.name = "NpuOptimizer"
custom_op.parameter_map["dynamic_input"].b = True
custom_op.parameter_map["dynamic_graph_execute_mode"].s = tf.compat.as_bytes("lazy_recompile")