python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1860048 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 7614>200 please execute split_portfolio.py -i 23315442 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2947888'] with mtr_portfolio_ids : ['23315442'] and first list_photo_ids : [] new path : /proc/1860048/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 47.33982062339783 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Tue May 27 19:51:15 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-05-27 19:51:18.658268: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-27 19:51:18.686519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-27 19:51:18.686821: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 19:51:18.688878: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 19:51:18.690636: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-27 19:51:18.690999: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-27 19:51:18.692995: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-27 19:51:18.694068: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-27 19:51:18.698292: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 19:51:18.699622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-27 19:51:18.699977: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-27 19:51:18.727241: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-27 19:51:18.729287: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7244000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-27 19:51:18.729342: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-27 19:51:18.840755: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2b2a7270 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-27 19:51:18.840802: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-27 19:51:18.842077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-27 19:51:18.842152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 19:51:18.842181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 19:51:18.842208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-27 19:51:18.842235: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-27 19:51:18.842261: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-27 19:51:18.842287: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-27 19:51:18.842314: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 19:51:18.844099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-27 19:51:18.844176: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 19:51:18.845204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-27 19:51:18.845228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-27 19:51:18.845243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-27 19:51:18.847105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 6997 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.7445650100708 time used to load_weights : 0.3095886707305908 0it [00:00, ?it/s]temp/1748368228_1860048_1360998445_39f52df2da3ab9f21e6c527e52a66f4e.jpg temp/1748368228_1860048_1360998443_95c9dae1e43e89a39e895f2077d3962f.jpg temp/1748368228_1860048_1360998441_37c5e0452a07a61c42ffd303c64122cf.jpg temp/1748368228_1860048_1360998439_f3b0ffa9d1eddc8bbf9707b481ffd74f.jpg temp/1748368228_1860048_1360998437_98ffe29a6d8943ba8afc5d74d5c09042.jpg temp/1748368228_1860048_1360998436_b259543db3a585fa49cd7866a6f5bc4d.jpg temp/1748368228_1860048_1360998423_dc51d28a59479d0668cbb15c7f80895d.jpg temp/1748368228_1860048_1360998421_17a323de2f513197fd03bfd2d5f99355.jpg temp/1748368228_1860048_1360998420_e19a35d29670c9281739fb1def742009.jpg temp/1748368228_1860048_1360998419_d592841ae2bd5790a3a78af53a86a5ea.jpg temp/1748368228_1860048_1360998418_596ad3e2370633e25c307efab8eba152.jpg temp/1748368228_1860048_1360998416_475e2e99ada82c68a79a84bc047e0900.jpg temp/1748368228_1860048_1360998389_de07d4eff2927e6e45b4322edd3c854d.jpg temp/1748368228_1860048_1360998387_8c1d405b48bfcb9090f6c9b7b800e882.jpg temp/1748368228_1860048_1360998385_4b7b9b42d90623ef537b46a366de5aa3.jpg temp/1748368228_1860048_1360998382_410d2869c8591557624b3c239567a128.jpg temp/1748368228_1860048_1360998379_d094a3599d14dba809c32be3c726c369.jpg temp/1748368228_1860048_1360998377_639565ec8cfd4191fd1e6f6b3df29b8a.jpg temp/1748368228_1860048_1360998374_b7243965b72987108e07663257836031.jpg temp/1748368228_1860048_1360998371_81bf2aee91171edc1514d79eb4e478b7.jpg temp/1748368228_1860048_1360998369_895b5c9c167e2dd59cb3358cccd304b4.jpg temp/1748368228_1860048_1360998368_50be462b7b91db636e075218659a09b6.jpg temp/1748368228_1860048_1360998367_08f0160672f59893dbb844b38d8897ac.jpg temp/1748368228_1860048_1360998366_79e29bf470f4e97950aafc9fd339510f.jpg temp/1748368228_1860048_1360998352_2c18661919b665455de009918a19fec4.jpg temp/1748368228_1860048_1360998350_9e2623391a3da226635d74e601ace32f.jpg temp/1748368228_1860048_1360998348_87f8d875e7e05440a69e8e1d3d03696b.jpg temp/1748368228_1860048_1360998347_de6059de45d04d47c852a11961bef147.jpg temp/1748368228_1860048_1360998345_e21aa6c74706c35181aa6cb0619b4418.jpg temp/1748368228_1860048_1360998343_24b0ed853ebe802fedd8aceab8160f98.jpg temp/1748368228_1860048_1360998330_48664f0d5d0bbfb814b4698ade6e66ae.jpg temp/1748368228_1860048_1360998328_72b0e9a5d8d073a69e1952e7c129353d.jpg temp/1748368228_1860048_1360998327_602bb415e6a34f228102001925f38100.jpg temp/1748368228_1860048_1360998326_d7859b79e3e814f2ebff9fed321fe449.jpg temp/1748368228_1860048_1360998325_c95c96b5a4c9b6448d0a5eba6aeabc23.jpg temp/1748368228_1860048_1360998324_3709275175eef91c90eac871cb4a0a7d.jpg temp/1748368228_1860048_1360998313_33bb702a3018b5bab420cef3920bb844.jpg temp/1748368228_1860048_1360998311_be50f358c780c91cc93e42f2c7a4e6fb.jpg temp/1748368228_1860048_1360998306_1a61b536e617d333d6f5092bf3e07c78.jpg temp/1748368228_1860048_1360998303_30a2ef1d81e85f8b0be2f56129f1782e.jpg temp/1748368228_1860048_1360998301_f212d94774832226c198ad1f03f7782f.jpg temp/1748368228_1860048_1360998299_dbd7af048e506c9d2d9b523b938d2a06.jpg temp/1748368228_1860048_1360998286_309050b767a51c8003de60ceb63fa762.jpg temp/1748368228_1860048_1360998282_16160c0a1c4dc881b94a5bb854cb358e.jpg temp/1748368228_1860048_1360998279_f31e44945d6c148aafe7f1578a07ebeb.jpg temp/1748368228_1860048_1360998275_ef9bd1cae76fab439fb5fad39ead262f.jpg temp/1748368228_1860048_1360998274_e23f82cabde8117d97d686010a4019ac.jpg temp/1748368228_1860048_1360998271_cc59cc5632d618fbd0ec7396c25df3aa.jpg temp/1748368228_1860048_1360998258_582131ce253638ab1bb7ead40e94d073.jpg temp/1748368228_1860048_1360998255_7e4578887e44dfa4d9404f63c8e90f8e.jpg temp/1748368228_1860048_1360998252_a0d57d0ddb1b7230ebd5bfc13db6f4cb.jpg temp/1748368228_1860048_1360998250_eff7a04d71fa48067fb30c90614c5737.jpg temp/1748368228_1860048_1360998248_06560e7cc028183d50978635245321e3.jpg temp/1748368228_1860048_1360998247_2ed5d98728d138d25391d7a9258861f5.jpg temp/1748368228_1860048_1360998233_49831f6b2e4f99d87df254c860082757.jpg temp/1748368228_1860048_1360998230_f373a4cd09aa393b90ca932f3046489f.jpg temp/1748368228_1860048_1360998228_02b7e5d2ee8b10f8a7659a2e2b197799.jpg temp/1748368228_1860048_1360998227_b2923a591e95dfffcdfd8fd5cfeddb3b.jpg temp/1748368228_1860048_1360998226_522e7baa780d0d93876eb695f35ff64d.jpg temp/1748368228_1860048_1360998224_aa67e736d821b8f50bf8235b13c874f1.jpg temp/1748368228_1860048_1360998207_459d1758d8fc37250faa99f380baae43.jpg temp/1748368228_1860048_1360998204_a623febe2334f80cd215131e2ec7d5aa.jpg temp/1748368228_1860048_1360998202_334885e05e5e814bc30d0a27fbea2108.jpg temp/1748368228_1860048_1360998201_ff68c99a83f7eb4815d3bc1a7a91ae8f.jpg temp/1748368228_1860048_1360998200_f2e091c0d1dc3f1d21c385f9374173d8.jpg temp/1748368228_1860048_1360998198_f0937903f5ab58f516454ba982f3466f.jpg temp/1748368228_1860048_1360998185_f8e4dc9888dad6b7b9d5d774d89f133c.jpg temp/1748368228_1860048_1360998183_e242a5c8808c4bc5168f1b0a9a71c787.jpg temp/1748368228_1860048_1360998181_c178361ab68bcdc36eda62a044f33d12.jpg temp/1748368228_1860048_1360998179_d9c13d84e1e0b1b6b9fa908b619215a7.jpg temp/1748368228_1860048_1360998177_c4e78fc12e8395461f6b57e340cc3dbf.jpg temp/1748368228_1860048_1360998176_0fed61048f35b68637f7d364b00f6892.jpg temp/1748368228_1860048_1360998163_6858c4b99f7a39e65c5865517e97da9f.jpg temp/1748368228_1860048_1360998161_d24c1d6e72442ba065b125f34118a699.jpg temp/1748368228_1860048_1360998159_cb1ce977e291e942e3e3f5b94b3fc07b.jpg temp/1748368228_1860048_1360998158_a44381e28af69c0303a53d296e12a8b4.jpg temp/1748368228_1860048_1360998157_56d1cc4e60aa9d9fe75e8fdb20adbe68.jpg temp/1748368228_1860048_1360998155_05b1cab5decbd7b9b4e4d14bab0f521d.jpg temp/1748368228_1860048_1360998142_b949684b23f3654eece7fc8ef87457e7.jpg temp/1748368228_1860048_1360998139_4e39c14f0158b981f366e3b124852fdb.jpg temp/1748368228_1860048_1360998136_e19681f16b8f915f073afa04c619d6f8.jpg temp/1748368228_1860048_1360998134_4c898d7d1534ddc9d121b327d646f190.jpg temp/1748368228_1860048_1360998132_a66a1e5bba4285f7fb9fc889119162c0.jpg temp/1748368228_1860048_1360998130_24abc2fb32c174eb6a2c4d4d051ace81.jpg temp/1748368228_1860048_1360998101_cdfce01bcfd4c2c1edc159a0e6a38f00.jpg temp/1748368228_1860048_1360998098_d4594f423e5e260c652d51cc903fef33.jpg temp/1748368228_1860048_1360998095_24b506e0fc898f977ce3b43aafa297e0.jpg temp/1748368228_1860048_1360998093_9c72c3f0787826df3ccd4e4822921bb0.jpg temp/1748368228_1860048_1360998091_73403620d5fd02cea3d90cd60005ce39.jpg temp/1748368228_1860048_1360998088_638c418b7b85ae47ae5e99bb249968c3.jpg temp/1748368228_1860048_1360998026_0f2a1449fabc1237cdedad7cdf65cd98.jpg temp/1748368228_1860048_1360998024_2e5c275e3ba7de4acc81c1901c2e0525.jpg temp/1748368228_1860048_1360998022_2144d764ba5cd5aa245d86468b18ba20.jpg temp/1748368228_1860048_1360998019_a1e6f2f00049710192fe0b6000ac5a6f.jpg temp/1748368228_1860048_1360998017_d3c50d39ecf3362333c7ba8414387f3e.jpg temp/1748368228_1860048_1360998015_c8cd389d2cc1333ce44bbffe8e96ab56.jpg temp/1748368228_1860048_1360998010_6dc006bbd1ab7b209b0cf431e2d2b60c.jpg temp/1748368228_1860048_1360998007_0bce3b064add4c7e08672f80e3c84cdd.jpg temp/1748368228_1860048_1360998005_0354d3281c3d65e3360ecd87579c432d.jpg temp/1748368228_1860048_1360998004_0d97ea299301b2dd7f0d1b6f32fb5280.jpg temp/1748368228_1860048_1360998003_5dd783e153117ade5d575165b90f727c.jpg temp/1748368228_1860048_1360998001_3c357c48b16c5bfd936c6aee7e41d0c3.jpg temp/1748368228_1860048_1360997992_3e9257ebedafd7bfe5152150eb929496.jpg temp/1748368228_1860048_1360997989_8855062e03057fb2ccfc9dc2286d3883.jpg temp/1748368228_1860048_1360997985_1577d8a9934346351386cb021b48ce7f.jpg temp/1748368228_1860048_1360997983_d792fa579366f9d89b6f4e87e8996d62.jpg temp/1748368228_1860048_1360997981_1dde13eb2d20782ebf05ab67ded2037b.jpg temp/1748368228_1860048_1360997980_e5a9e86452ec48fb474d1287501a79f2.jpg temp/1748368228_1860048_1360997963_8fc666917da5b9a4e929664dfb1e7b6a.jpg temp/1748368228_1860048_1360997960_076daefb92d229a6e32bd7516bc4b6da.jpg temp/1748368228_1860048_1360997958_76d622feaffdfec033b4f57bc0068130.jpg temp/1748368228_1860048_1360997957_d93b8aafea7d235be75e6aa5025d4898.jpg temp/1748368228_1860048_1360997956_f6e043cf27db3265c34e9d436bbb0515.jpg 172it [00:00, 1716.31it/s] 200it [00:00, 1711.74it/s]2025-05-27 19:51:30.052653: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 19:51:30.294798: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-27 19:51:30.323520: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 19:51:30.335924: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-27 19:51:30.339006: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748368228_1860048_1360997954_7ca20792ddde4357dabb682fbd56e784.jpg temp/1748368228_1860048_1360997927_3dd33581d487b1088baa6575a1bd4a5c.jpg temp/1748368228_1860048_1360997925_3cc6f3aaa4a0e9cdee32e8c2ef16e14f.jpg temp/1748368228_1860048_1360997923_788ca6dc4139e90476a2110109b1bb7a.jpg temp/1748368228_1860048_1360997920_49c0f7c28e044f051984591c5ef843c3.jpg temp/1748368228_1860048_1360997917_91e1625d129cb93945e081e68db113a6.jpg temp/1748368228_1860048_1360997915_a438841c63d4b910c568efed91e02551.jpg temp/1748368228_1860048_1360997910_5d1f90fb1307588fc29c303fb74a3a3f.jpg temp/1748368228_1860048_1360997907_039a9e94c13d7e2153fce3f254f5717a.jpg temp/1748368228_1860048_1360997905_e3c6f0e172c2f50f489871860f467d4c.jpg temp/1748368228_1860048_1360997904_e871255e77de0baab7e9c758ff57b305.jpg temp/1748368228_1860048_1360997903_0e615ec01562678eca57a11e21cbd284.jpg temp/1748368228_1860048_1360997901_b79da2167371ad490055278570e36357.jpg temp/1748368228_1860048_1360997886_0e2a5e0274bde4af621fa2807d36f1dd.jpg temp/1748368228_1860048_1360997883_d87ed810cf74088b97b1b0df63b763f9.jpg temp/1748368228_1860048_1360997880_042fbb8c110afb00608174cae760c15c.jpg temp/1748368228_1860048_1360997879_e64c6da44e518bd802b1249f7767146e.jpg temp/1748368228_1860048_1360997877_1d31e00e32d8ffc5f5c9fd88ef0c430f.jpg temp/1748368228_1860048_1360997875_0414852fd25170c8f6deb10ca8e0082f.jpg temp/1748368228_1860048_1360997858_948c052397a2f1341a34ff185a9f547a.jpg temp/1748368228_1860048_1360997855_c87c54d6c4b8c8225a00d4bffb1ca051.jpg temp/1748368228_1860048_1360997853_2e62d950424865603038befd2ed7789d.jpg temp/1748368228_1860048_1360997852_aa0dd6f806366d09bd788fb34e0c979b.jpg temp/1748368228_1860048_1360997851_51bb5aaabb4b52b19aba6edaf5f72955.jpg temp/1748368228_1860048_1360997849_03ed29301278109eb63460c4467ada59.jpg temp/1748368228_1860048_1360997836_5aedb6e1a93807d0ec1cc25a0f26974a.jpg temp/1748368228_1860048_1360997833_f5377954e7cd636a9f43ed28c824c833.jpg temp/1748368228_1860048_1360997830_644a4b2ec82df025281a9f1b1e0d5fec.jpg temp/1748368228_1860048_1360997827_57780e2aa7c1437ab6c5c419b1b35703.jpg temp/1748368228_1860048_1360997825_743dce5834a58bac9b381e55c0c2c2ef.jpg temp/1748368228_1860048_1360997823_9009b90c5c25fae73188551c8571ded9.jpg temp/1748368228_1860048_1360997809_d506a4686096ad98fcae9a3f08778005.jpg temp/1748368228_1860048_1360997805_f0ad3bd8a0a2b02191501bb4d246504b.jpg temp/1748368228_1860048_1360997801_7d1e0a5d8e74d48f3a61e81dd6ae9f2d.jpg temp/1748368228_1860048_1360997799_4e65027dae6beb726cd05b3bde34a985.jpg temp/1748368228_1860048_1360997797_bc431b80620765e51a3753b0d8522cf6.jpg temp/1748368228_1860048_1360997796_4c3e0c9cdd58a2e53ef8e0c4aa7f1c43.jpg temp/1748368228_1860048_1360997781_aa490bab2b88365527caee2ec5c38075.jpg temp/1748368228_1860048_1360997777_49240f2ff0a0f4683a79f894b6cc92d6.jpg temp/1748368228_1860048_1360997774_28a537e6df15c28e0e4dc497e3841a04.jpg temp/1748368228_1860048_1360997771_fad10418c6d30ed58f6047e558d118b1.jpg temp/1748368228_1860048_1360997769_77e1466ae6b31e5c1b2201557dce8a2f.jpg temp/1748368228_1860048_1360997766_298ff971c1af7b5d7818e650dc69661e.jpg temp/1748368228_1860048_1360997753_4845269ac11ad4d407e2aad096f23c31.jpg temp/1748368228_1860048_1360997750_b1ee3dc0993517b5c3f5fbe122a996f2.jpg temp/1748368228_1860048_1360997747_056786c2300fd48afc0e1839ce3ee3a8.jpg temp/1748368228_1860048_1360997745_da524c8a8fdc9b96b491e48d3b294cb1.jpg temp/1748368228_1860048_1360997744_0c6cf9c170368837371128314b39caae.jpg temp/1748368228_1860048_1360997742_90022981630f729bf0aef0c38fd0f5c8.jpg temp/1748368228_1860048_1360997725_e6a6b662d5d4910aed1e0ab1b592e872.jpg temp/1748368228_1860048_1360997722_ff55006b88cea3da16e34c962958f49a.jpg temp/1748368228_1860048_1360997720_156b37b42a73c18a00d848cbe88d753f.jpg temp/1748368228_1860048_1360997719_74177de24c90045ea3b504e5271836f9.jpg temp/1748368228_1860048_1360997718_9f41ca6070f906b3b8aca45c3d0f00c0.jpg temp/1748368228_1860048_1360997716_eb8238d2f3c8daa7e07dfc39a0002bec.jpg temp/1748368228_1860048_1360997701_6c2049e257c9074da7dcc3280bed533b.jpg temp/1748368228_1860048_1360997699_2d16190045ee50c380728754cbbc4e75.jpg temp/1748368228_1860048_1360997696_a323192cf063800cb226cc332e74d872.jpg temp/1748368228_1860048_1360997694_2b05cbc5c0aa6c3b24d29b1a25d5e652.jpg temp/1748368228_1860048_1360997692_6006e50b165c461e4edd882e8af309e2.jpg temp/1748368228_1860048_1360997690_5542df71060db98a6184a211deafcd35.jpg temp/1748368228_1860048_1360997677_a789ae58efd2855bd727e42eef417075.jpg temp/1748368228_1860048_1360997674_48f58d8f456108b9a1ad89501d509741.jpg temp/1748368228_1860048_1360997672_57a0c014f94158694f772e75adffbccd.jpg temp/1748368228_1860048_1360997671_55ef4be003cb28b1327a937efedc852e.jpg temp/1748368228_1860048_1360997670_d7c270bbb55d3e65bc137e414885a1ea.jpg temp/1748368228_1860048_1360997668_14eeac50562ce1ffc1a7cc43fc382813.jpg temp/1748368228_1860048_1360997618_492f8c85485220c1e5c4ea173592d78b.jpg temp/1748368228_1860048_1360997615_3e7acdd025f0862408d4ca008060726a.jpg temp/1748368228_1860048_1360997612_998e6d60ced18d8adab282ed3412a536.jpg temp/1748368228_1860048_1360997609_2c1b2a7a6160f7190594d62eb087aebf.jpg temp/1748368228_1860048_1360997607_38b791704b7dd5ea7ef463551457cd71.jpg temp/1748368228_1860048_1360997606_d72d4c39705689b767f945b2e2c8fa83.jpg temp/1748368228_1860048_1360997603_a5a121248cc0a0efbbd58b66bff94be8.jpg temp/1748368228_1860048_1360997600_7df1df962948c804fb2b4ddbc2fbcc9d.jpg temp/1748368228_1860048_1360997597_603891d1e9c141d681998d499e1c6b86.jpg temp/1748368228_1860048_1360997595_e611ed10482222ef242e5b3ca7b4ef4c.jpg temp/1748368228_1860048_1360997593_fc1ec1888e4f0a27846eb03db1c5d997.jpg temp/1748368228_1860048_1360997592_7746b07313df132a070bdf92bcd44c22.jpg temp/1748368228_1860048_1360997515_7e27cede2b923459a4b2ecc95f75a5cf.jpg temp/1748368228_1860048_1360997512_67bf8110ef109f1166c87925b686e933.jpg temp/1748368228_1860048_1360997510_f3b9e65e10c26051fb2b3a6304c50cbd.jpg temp/1748368228_1860048_1360997509_8d6c5178271626ae636dc6d5964ec242.jpg temp/1748368228_1860048_1360997508_600a883d86cd3b658aeb1d418eddba56.jpg temp/1748368228_1860048_1360997507_e8dfe552e182d556b1a134eff7eb02c0.jpg temp/1748368228_1860048_1360997475_443f98da7bef24c7bddbf198740ae80a.jpg temp/1748368228_1860048_1360997472_d9520b24a40c8fb0795ab50eb767f03e.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1360998445, 1360998443, 1360998441, 1360998439, 1360998437, 1360998436, 1360998423, 1360998421, 1360998420, 1360998419, 1360998418, 1360998416, 1360998389, 1360998387, 1360998385, 1360998382, 1360998379, 1360998377, 1360998374, 1360998371, 1360998369, 1360998368, 1360998367, 1360998366, 1360998352, 1360998350, 1360998348, 1360998347, 1360998345, 1360998343, 1360998330, 1360998328, 1360998327, 1360998326, 1360998325, 1360998324, 1360998313, 1360998311, 1360998306, 1360998303, 1360998301, 1360998299, 1360998286, 1360998282, 1360998279, 1360998275, 1360998274, 1360998271, 1360998258, 1360998255, 1360998252, 1360998250, 1360998248, 1360998247, 1360998233, 1360998230, 1360998228, 1360998227, 1360998226, 1360998224, 1360998207, 1360998204, 1360998202, 1360998201, 1360998200, 1360998198, 1360998185, 1360998183, 1360998181, 1360998179, 1360998177, 1360998176, 1360998163, 1360998161, 1360998159, 1360998158, 1360998157, 1360998155, 1360998142, 1360998139, 1360998136, 1360998134, 1360998132, 1360998130, 1360998101, 1360998098, 1360998095, 1360998093, 1360998091, 1360998088, 1360998026, 1360998024, 1360998022, 1360998019, 1360998017, 1360998015, 1360998010, 1360998007, 1360998005, 1360998004, 1360998003, 1360998001, 1360997992, 1360997989, 1360997985, 1360997983, 1360997981, 1360997980, 1360997963, 1360997960, 1360997958, 1360997957, 1360997956, 1360997954, 1360997927, 1360997925, 1360997923, 1360997920, 1360997917, 1360997915, 1360997910, 1360997907, 1360997905, 1360997904, 1360997903, 1360997901, 1360997886, 1360997883, 1360997880, 1360997879, 1360997877, 1360997875, 1360997858, 1360997855, 1360997853, 1360997852, 1360997851, 1360997849, 1360997836, 1360997833, 1360997830, 1360997827, 1360997825, 1360997823, 1360997809, 1360997805, 1360997801, 1360997799, 1360997797, 1360997796, 1360997781, 1360997777, 1360997774, 1360997771, 1360997769, 1360997766, 1360997753, 1360997750, 1360997747, 1360997745, 1360997744, 1360997742, 1360997725, 1360997722, 1360997720, 1360997719, 1360997718, 1360997716, 1360997701, 1360997699, 1360997696, 1360997694, 1360997692, 1360997690, 1360997677, 1360997674, 1360997672, 1360997671, 1360997670, 1360997668, 1360997618, 1360997615, 1360997612, 1360997609, 1360997607, 1360997606, 1360997603, 1360997600, 1360997597, 1360997595, 1360997593, 1360997592, 1360997515, 1360997512, 1360997510, 1360997509, 1360997508, 1360997507, 1360997475, 1360997472] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.09675979614257812 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 14.397676229476929 time spend to save output : 0.10419273376464844 total time spend for step 0 : 14.501868963241577 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.69user 4.05system 1:06.49elapsed 28%CPU (0avgtext+0avgdata 1844964maxresident)k 2272inputs+314480outputs (0major+485606minor)pagefaults 0swaps