python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2162056 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 5724>200 please execute split_portfolio.py -i 22530011 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2772389'] with mtr_portfolio_ids : ['22530011'] and first list_photo_ids : [] new path : /proc/2162056/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 26.974759340286255 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu May 1 13:00:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1990 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1990 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1990 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1990 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1990 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1990 wait 20 seconds 2025-05-01 13:03:03.747959: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-01 13:03:03.778556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-01 13:03:03.778836: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-01 13:03:03.781111: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-01 13:03:03.783187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-01 13:03:03.783560: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-01 13:03:03.785768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-01 13:03:03.786804: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-01 13:03:03.791071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-01 13:03:03.792194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-01 13:03:03.792546: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-01 13:03:03.799919: I tensor2025-05-01 13:03:10.378395: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-01 13:03:10.379018: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-01 13:03:10.379604: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-01 13:03:10.380177: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.20G (2366618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-01 13:03:10.380707: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.98G (2129957120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-01 13:03:10.381239: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.79G (1916961536 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.927034139633179 time used to load_weights : 0.21516990661621094 0it [00:00, ?it/s]temp/1746097229_2162056_1355094666_6eaa3491ad48c6e9706e8299034b0ea9.jpg temp/1746097229_2162056_1355094664_7dbaa72891596b60ff3fe47ab0fd7fa4.jpg temp/1746097229_2162056_1355094663_9bab38720ac7b7dc277831bb3c104a3d.jpg temp/1746097229_2162056_1355094662_3fa1442422e90e3980c9a26586eabc3b.jpg temp/1746097229_2162056_1355094661_e35aaa61fb877dfc3aa331359b9c7971.jpg temp/1746097229_2162056_1355094659_f462a7bb0241bcb2cec1a352d287b475.jpg temp/1746097229_2162056_1355094620_3552359b624481292613a294d649886a.jpg temp/1746097229_2162056_1355094617_f058955f413337d724ae4def60b8c1f9.jpg temp/1746097229_2162056_1355094616_8309ae85558903a3807b22da80f9c20c.jpg temp/1746097229_2162056_1355094614_3e4ae693a5eb8b890675de33da09eb02.jpg temp/1746097229_2162056_1355094610_b97590200ee1a69ec38ce80a6013976e.jpg temp/1746097229_2162056_1355094607_947c48980f421dc5d1648cbb7ddc7957.jpg temp/1746097229_2162056_1355094556_1a2b7a44ae5f4d330b81aa535417355b.jpg temp/1746097229_2162056_1355094554_bc9d10757c2db86cc0b3b2ecad6feb0e.jpg temp/1746097229_2162056_1355094530_4ccbe0f78be122946b8a9cc63928f584.jpg temp/1746097229_2162056_1355094245_89ebbac6a9d5cafaa4cffd114ce26073.jpg temp/1746097229_2162056_1355093986_215cc8e3ddc93f0a07e127620aa263c4.jpg temp/1746097229_2162056_1355093984_22a8a44cdaf7a0c67118b0485d8fc1df.jpg temp/1746097229_2162056_1355093971_ba9476210197b467ad8f2e3faa75578f.jpg temp/1746097229_2162056_1355093967_b6fc8dcf7c5d9ddec911e7b82c54d750.jpg temp/1746097229_2162056_1355093963_4e32f15298fce5ae91a1141d080b8c69.jpg temp/1746097229_2162056_1355093958_4b1da780a44de388f8c674c758fd7a49.jpg temp/1746097229_2162056_1355093955_444874797a7a28139b4ed5a2897b0638.jpg temp/1746097229_2162056_1355093953_6614a019bd19030e6c6b8eb8393b8e1a.jpg temp/1746097229_2162056_1355093940_94e2c1af769666ae42e8c8ad6513bbcc.jpg temp/1746097229_2162056_1355093938_9f4a010ae2bbb48c4c978e2286ad8208.jpg temp/1746097229_2162056_1355093935_21e54cbafb49082fcb479f3e9f64fdf0.jpg temp/1746097229_2162056_1355093932_d65ac96f6c7a6c04d3ddbf38b2ca85cd.jpg temp/1746097229_2162056_1355093928_968c7bc91fe33566ffa99065a15c688f.jpg temp/1746097229_2162056_1355093926_27ebc7bc6c5836f2969bf6548c669e95.jpg temp/1746097229_2162056_1355093912_18722d7a2a3ca52b4499b660a7548dff.jpg temp/1746097229_2162056_1355093910_b790d8434db266ab16dd061337586674.jpg temp/1746097229_2162056_1355093905_fca9076f9fa3763a4f29d60d34de3e4e.jpg temp/1746097229_2162056_1355093903_856231be634c76af6869e4deb797204f.jpg temp/1746097229_2162056_1355093899_47bb8cb27f5f9fda64ec1d52fab4107e.jpg temp/1746097229_2162056_1355093897_b0d3cbf26752f90f9bd6cfca8c5ebd6f.jpg temp/1746097229_2162056_1355093858_99ca5afa0b4528f95cfd92baba9cbc71.jpg temp/1746097229_2162056_1355093855_3f351e302e4cf3e3f516f0eb5c75982a.jpg temp/1746097229_2162056_1355093851_c4959acf40d4bd9204f9f8c28c7e801c.jpg temp/1746097229_2162056_1355093844_04eba6e2d4d638b2977b10d7df1b6db4.jpg temp/1746097229_2162056_1355093839_ef9a5f09c270ea8b8eef358402a499dd.jpg temp/1746097229_2162056_1355093838_38615585a277c933ea7b2591dd873413.jpg temp/1746097229_2162056_1355093829_5e4a6a6d537032a5cb273ebb1dc6a425.jpg temp/1746097229_2162056_1355093825_56c28d457689313d8d4a03a5744184fb.jpg temp/1746097229_2162056_1355093821_14cd6b4a4e940751d7b4f77839df7785.jpg temp/1746097229_2162056_1355093817_210308e03c1644f60b6fbd69913a36a1.jpg temp/1746097229_2162056_1355093810_d424a458819fcbac781764f3630b4cbb.jpg temp/1746097229_2162056_1355093807_15f79f68bdaf5fe522b84239574dc705.jpg temp/1746097229_2162056_1355093775_8f2bbcf9cea3a9d0830b390eea7672c1.jpg temp/1746097229_2162056_1355093773_0173044e37744bd9c8ce4c34bb3ba913.jpg temp/1746097229_2162056_1355093771_62876e7b47693f17f2a75d9eb71d520d.jpg temp/1746097229_2162056_1355093768_ace3aafe2a49a0e21faff7a7edd1e2c8.jpg temp/1746097229_2162056_1355093764_47092b25dbb7a599a7b758b2878b36cf.jpg temp/1746097229_2162056_1355093760_a532afaf7f802e25075882fe8152d609.jpg temp/1746097229_2162056_1355093733_4d9cf6f556645e5a8fa55b2ad96c2352.jpg temp/1746097229_2162056_1355093730_82ceb6afa399432537bc67a05518ddbe.jpg temp/1746097229_2162056_1355093725_5136799e39ac581124f92b57ce0b9317.jpg temp/1746097229_2162056_1355093722_74ed7a58571836375a0c644b76043d9f.jpg temp/1746097229_2162056_1355093720_f45b72c655802fe7a57887f90f7718b1.jpg temp/1746097229_2162056_1355093718_7b487f7382b2f00d869caf5fb63155f2.jpg temp/1746097229_2162056_1355093676_82073d34f657e2b376d5e3c95dde0459.jpg temp/1746097229_2162056_1355093670_113069a11a0af9a001069bed95725d05.jpg temp/1746097229_2162056_1355093665_18cdbba078138d8b82cd8322202f434d.jpg temp/1746097229_2162056_1355093662_b68b8a5daaee0c9c437243d14f8d9c94.jpg temp/1746097229_2162056_1355093659_8efc68751711a5c4d26ef475eb75136c.jpg temp/1746097229_2162056_1355093658_4ec72d2b308fa26155da5b120f45f2f6.jpg temp/1746097229_2162056_1355093652_cc6a8dab661bb869ff2b77bd131ef4e7.jpg temp/1746097229_2162056_1355093648_3c778782b9d3634a97d3b19e7a6d409b.jpg temp/1746097229_2162056_1355093643_eff8e6be90388782441366b1b0267edd.jpg temp/1746097229_2162056_1355093638_4bf99a4a5cc4320a518ad7d01d3b2f85.jpg temp/1746097229_2162056_1355093629_d3938f20098b7cc829e40896c498c429.jpg temp/1746097229_2162056_1355093625_522997eecd8c00ea191b1338987f3e18.jpg temp/1746097229_2162056_1355093608_2089da96eedba6edcbd403ab714a638a.jpg temp/1746097229_2162056_1355093604_4479dc25896b4a476710c35ed972c390.jpg temp/1746097229_2162056_1355093601_f60c7b99f5ba20c383bcec2ee0a681db.jpg temp/1746097229_2162056_1355093598_b3d3fc149cb79cca4b64ec645c5f2850.jpg temp/1746097229_2162056_1355093597_87398fc640092fe2a817b5e76b525254.jpg temp/1746097229_2162056_1355093595_caf69c47abf3276056a2a2e9e7fe607d.jpg temp/1746097229_2162056_1355093582_900c549e00cbf20be4ea03499f400d82.jpg temp/1746097229_2162056_1355093558_f80100d626c34af488f2b679d5941a7b.jpg temp/1746097229_2162056_1355093528_8617ca98cf4a691a33b1764196e3089b.jpg temp/1746097229_2162056_1355093512_ff76925964e7288230c372d29f16c2e1.jpg temp/1746097229_2162056_1355093508_41b5f807095b427c51f3a35099a635fa.jpg temp/1746097229_2162056_1355093505_72d48451e439c468e1b41856e3f10d1f.jpg temp/1746097229_2162056_1355093399_f5a883f68e81f1037fda234ffc9fd8c5.jpg temp/1746097229_2162056_1355093395_a3dc0753f89e33256e849356ffd06780.jpg temp/1746097229_2162056_1355093390_c0674d5e5700c77f42cb102990e3388b.jpg temp/1746097229_2162056_1355093388_770ac55defb8ba69812a3b3f49651db1.jpg temp/1746097229_2162056_1355093383_0006ccb937cf012f333537c8679a7fd3.jpg temp/1746097229_2162056_1355093381_9983b339499dcb3f997530de0fc6b97e.jpg temp/1746097229_2162056_1355093354_83f01ba91b1ab53efa84227ac80cbfb0.jpg temp/1746097229_2162056_1355093351_f6874517961fda3f5742c1f1ff8d1c4f.jpg temp/1746097229_2162056_1355093349_c116b5b0b15adbba85b14c30c97096ed.jpg temp/1746097229_2162056_1355093345_c6fb356558adb910a11586db34c2c520.jpg temp/1746097229_2162056_1355093341_3603154a56233e3e8412bf41f7ec4ba6.jpg temp/1746097229_2162056_1355093337_440d1e032d8f5200eea1f9f794f0b9c3.jpg temp/1746097229_2162056_1355093317_eb5b103e985a9909cb463537f3f3d28d.jpg temp/1746097229_2162056_1355093312_c62b9b723fe2996506c0f57fc4660639.jpg temp/1746097229_2162056_1355093307_a1d73da54916067c83429ee7df12b889.jpg temp/1746097229_2162056_1355093304_6d471ff969d351e9504519d4c2ce2837.jpg temp/1746097229_2162056_1355093300_c83fadff85ea5ca2449cca64fc8e4ec1.jpg temp/1746097229_2162056_1355093298_9fce5ecc6068a677b92d29689ec999d9.jpg temp/1746097229_2162056_1355093263_9e7ac40a88d5873ed6ff656d5af70223.jpg temp/1746097229_2162056_1355093261_db5ef0d31ccf31a237006a40cb39cbf7.jpg temp/1746097229_2162056_1355093258_3566dcaf85e94ad10cfc755c8e8c67b3.jpg temp/1746097229_2162056_1355093255_d869ca8c89b52c02f1e6a6abaaa2fac9.jpg temp/1746097229_2162056_1355093252_940090041272291860b4f06a0e11c3ef.jpg temp/1746097229_2162056_1355093248_17b105eb5207433e6af5e193a3d22a0f.jpg temp/1746097229_2162056_1355093220_c35600171c573153ec4368005f4c4391.jpg temp/1746097229_2162056_1355093217_65dd568b8f2213c9f76ec8eb268c95c5.jpg temp/1746097229_2162056_1355093213_3e38b82b11b570d3cdaf91dda683f32f.jpg temp/1746097229_2162056_1355093211_0d8bd08eae49d3a2dccaaba6f757caa7.jpg temp/1746097229_2162056_1355093205_9ba79f2695e6fbc3fd9985cc9db1285a.jpg 181it [00:00, 1802.23it/s] 200it [00:00, 1807.24it/s]2025-05-01 13:03:14.100402: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-01 13:03:14.169478: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-01 13:03:14.198453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-01 13:03:14.203252: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-01 13:03:14.205048: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1746097229_2162056_1355093202_8f49e9c7a0d1331c9913bddca21a1f0f.jpg temp/1746097229_2162056_1355093117_f8bd87b0448da81aa58b1409dca7ea85.jpg temp/1746097229_2162056_1355093114_e01a6ef20f36565e9e3f11f9a32e0875.jpg temp/1746097229_2162056_1355093110_f081243595e47bb77f1e2a8830048276.jpg temp/1746097229_2162056_1355093106_c00b6b4bb7c06413b0bd79e0ca3e07ef.jpg temp/1746097229_2162056_1355093097_341763ea04b4cdf3fdf3893a08940926.jpg temp/1746097229_2162056_1355093086_59d4c9d55df184de1d09a40f85c7b98d.jpg temp/1746097229_2162056_1355093068_d5fb226ecd626de29aef8403a3b997a2.jpg temp/1746097229_2162056_1355093066_4b8af592574feee2f747a19e19eed9af.jpg temp/1746097229_2162056_1355093062_1b02780e517c1dc52668e89ed8c83e37.jpg temp/1746097229_2162056_1355093058_37cd652326175ac3543729343a711eda.jpg temp/1746097229_2162056_1355093054_3e386156b7b510a68789953b02bc7e1a.jpg temp/1746097229_2162056_1355093051_2138462a458c8287eb416e34a7dfb4a7.jpg temp/1746097229_2162056_1355093034_afb915d4f39e01a2aee3d1497de88d41.jpg temp/1746097229_2162056_1355093031_a0914a6ff1be97e186fed05a254e2ddf.jpg temp/1746097229_2162056_1355093027_bd5a2f3bcc16016cc8a5bade3a7e7e9d.jpg temp/1746097229_2162056_1355093025_e9f60495559c89390133defe738f9ad0.jpg temp/1746097229_2162056_1355093023_b6bbc0bd0b4a6145dc9a382ce3953b12.jpg temp/1746097229_2162056_1355093022_dee91ddee87e2d99a896980dd2e73a20.jpg temp/1746097229_2162056_1355092949_f4044705af880ba9e19a159ed80f40c1.jpg temp/1746097229_2162056_1355092945_f2eb29b9e1538c849368bb1c14ee6b92.jpg temp/1746097229_2162056_1355092941_838ba914080a39bfd1eced3540e1f14f.jpg temp/1746097229_2162056_1355092938_e88c2753ecdfb15808dd4ad79bb7d6e3.jpg temp/1746097229_2162056_1355092935_7ad49479b0c01df6c66064141f440683.jpg temp/1746097229_2162056_1355092934_e90b13fe393417618cc8a0ad69596ef8.jpg temp/1746097229_2162056_1355092920_573dccf25377d45eb6bebb38a412ddaa.jpg temp/1746097229_2162056_1355092914_663b8f409e4d5233389a50729f49a6db.jpg temp/1746097229_2162056_1355092911_74d4cbe702d8ee1bbfa678e31b5dbdf2.jpg temp/1746097229_2162056_1355092909_a735c86a1262ba37e6e1fa4e0c150393.jpg temp/1746097229_2162056_1355092905_0bf616a3013b17a497aa24632bc57bb5.jpg temp/1746097229_2162056_1355092904_a52bc0ad5fa8a080753f432fcd8cd17f.jpg temp/1746097229_2162056_1355092887_0445ff2b369825253293e9da9a553c79.jpg temp/1746097229_2162056_1355092884_8c35d36ca21df287f8136d9abd2c1482.jpg temp/1746097229_2162056_1355092881_98720e2dea7d8db6d5b4f05fcb1a9fc8.jpg temp/1746097229_2162056_1355092880_91e651b7b4f4306b1c2920198641a731.jpg temp/1746097229_2162056_1355092877_92fe01f55fdbb5e063f945878f833a96.jpg temp/1746097229_2162056_1355092876_214c8974521619344a0c955ab69bc90c.jpg temp/1746097229_2162056_1355092858_1a592cfb039cc43f88d5923e736d815a.jpg temp/1746097229_2162056_1355092855_b3c25a8cf2f9f0e85e7c955bd7d30524.jpg temp/1746097229_2162056_1355092852_9a48d305439e0d5675f08e70fb1e97cb.jpg temp/1746097229_2162056_1355092850_0a4cc8b174d6fe85975a600d6bbb010b.jpg temp/1746097229_2162056_1355092847_9e41b3e0e429449a47c371f73476a7ed.jpg temp/1746097229_2162056_1355092846_3b86ab288328f2293a0d7f15b65a96fa.jpg temp/1746097229_2162056_1355092828_b1d04955d50515acab4a1e2de34965c1.jpg temp/1746097229_2162056_1355092823_28869c6a274a986f901710419ac6da91.jpg temp/1746097229_2162056_1355092816_f75e6f9ce846a414d8be627679a21a1b.jpg temp/1746097229_2162056_1355092812_4b9366aee41a03cc64d6fe9cf2f2c6d9.jpg temp/1746097229_2162056_1355092809_f593588e79224021f39e5005025ea598.jpg temp/1746097229_2162056_1355092808_8ccbbd56449f052fa924e75110b6239b.jpg temp/1746097229_2162056_1355092738_1c0e6907c6dfe83e8baca9703c1dd63c.jpg temp/1746097229_2162056_1355092728_6fa8ca048261d82b9a1ba59814ad589d.jpg temp/1746097229_2162056_1355092713_5dc3f68b4f6801e589ba3a6a04fb452a.jpg temp/1746097229_2162056_1355092711_5940be8debbc047451b70a6d682454c1.jpg temp/1746097229_2162056_1355092706_e4b813502f18758526fa491027416416.jpg temp/1746097229_2162056_1355092704_0c8b46c040edeed299f2fa8f8ecb9ab5.jpg temp/1746097229_2162056_1355092687_95835c9f6b0fb46933e3b71645d5188a.jpg temp/1746097229_2162056_1355092683_1c9b0c396e2f5af86f12f811f824b5fe.jpg temp/1746097229_2162056_1355092678_c4ad0b2ee3b5fd2a8604c1118cca5719.jpg temp/1746097229_2162056_1355092674_0956cc579507b7f3103125773fb5dd75.jpg temp/1746097229_2162056_1355092672_6948eab5ec694e08ae923d4c171e4646.jpg temp/1746097229_2162056_1355092670_b5d6392738e20a25871d61e96f216cf4.jpg temp/1746097229_2162056_1355092653_caa7df3bbc162061fc11931bfeeb1627.jpg temp/1746097229_2162056_1355092649_f01498dabd155a5e0a5c1a892af85180.jpg temp/1746097229_2162056_1355092644_88048512b590fb2fc7efb4f8e2fb4cb9.jpg temp/1746097229_2162056_1355092641_35b7f8e9760de7ea9e3a3f488372e6c8.jpg temp/1746097229_2162056_1355092637_c30d186d8b544dd5b788c674c0cc1168.jpg temp/1746097229_2162056_1355092635_8680491026eb3c3792dea4f69c26bdd3.jpg temp/1746097229_2162056_1355092573_dce8dfcbdee4a43d6ebfd0b4c49217ad.jpg temp/1746097229_2162056_1355092570_56d16273d2d18683dbf47f48b113be08.jpg temp/1746097229_2162056_1355092567_75c55e74ce64948feb6b08a7eb313517.jpg temp/1746097229_2162056_1355092564_3ab38bfe3cc6b3c7b5df3a5fd3b4e51c.jpg temp/1746097229_2162056_1355092559_3e6380c8d64f7b90fb47d87bad1b1e7c.jpg temp/1746097229_2162056_1355092557_7179e5d2498f0e4ee53f55e2708aaffa.jpg temp/1746097229_2162056_1355092515_bf8ca1b12ac3442af2c1dd5b1fc3e8d8.jpg temp/1746097229_2162056_1355092511_0321bf634b6a974e1e1a874b88e7d1d5.jpg temp/1746097229_2162056_1355092507_358bc202c5baec9d8ae4f3f26d0971b1.jpg temp/1746097229_2162056_1355092503_9149ed78f3c5fbfd0a3075ee1b6bc175.jpg temp/1746097229_2162056_1355092500_4b09638e71162243d30bea8a55703703.jpg temp/1746097229_2162056_1355092498_39a562caa631a32b249ecac0a2c978e5.jpg temp/1746097229_2162056_1355092484_121ea81e8147c03bcd3aea387afb81fc.jpg temp/1746097229_2162056_1355092481_1b4047686b0dbfc887c90787cc8e8529.jpg temp/1746097229_2162056_1355092477_1ac3bf4b7013ebe88dea1a75bf120038.jpg temp/1746097229_2162056_1355092476_274c71f8e91c11386820294722e8076e.jpg temp/1746097229_2162056_1355092473_b9ba8f2e99a95db9756e9c2aff2cd44b.jpg temp/1746097229_2162056_1355092471_6e2a975f80842630beec6eae503ffaee.jpg temp/1746097229_2162056_1355092454_77769e7715c537314374215e4b5c3e66.jpg temp/1746097229_2162056_1355092452_80ddad61c7a85492050c577d40b551bd.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1355094666, 1355094664, 1355094663, 1355094662, 1355094661, 1355094659, 1355094620, 1355094617, 1355094616, 1355094614, 1355094610, 1355094607, 1355094556, 1355094554, 1355094530, 1355094245, 1355093986, 1355093984, 1355093971, 1355093967, 1355093963, 1355093958, 1355093955, 1355093953, 1355093940, 1355093938, 1355093935, 1355093932, 1355093928, 1355093926, 1355093912, 1355093910, 1355093905, 1355093903, 1355093899, 1355093897, 1355093858, 1355093855, 1355093851, 1355093844, 1355093839, 1355093838, 1355093829, 1355093825, 1355093821, 1355093817, 1355093810, 1355093807, 1355093775, 1355093773, 1355093771, 1355093768, 1355093764, 1355093760, 1355093733, 1355093730, 1355093725, 1355093722, 1355093720, 1355093718, 1355093676, 1355093670, 1355093665, 1355093662, 1355093659, 1355093658, 1355093652, 1355093648, 1355093643, 1355093638, 1355093629, 1355093625, 1355093608, 1355093604, 1355093601, 1355093598, 1355093597, 1355093595, 1355093582, 1355093558, 1355093528, 1355093512, 1355093508, 1355093505, 1355093399, 1355093395, 1355093390, 1355093388, 1355093383, 1355093381, 1355093354, 1355093351, 1355093349, 1355093345, 1355093341, 1355093337, 1355093317, 1355093312, 1355093307, 1355093304, 1355093300, 1355093298, 1355093263, 1355093261, 1355093258, 1355093255, 1355093252, 1355093248, 1355093220, 1355093217, 1355093213, 1355093211, 1355093205, 1355093202, 1355093117, 1355093114, 1355093110, 1355093106, 1355093097, 1355093086, 1355093068, 1355093066, 1355093062, 1355093058, 1355093054, 1355093051, 1355093034, 1355093031, 1355093027, 1355093025, 1355093023, 1355093022, 1355092949, 1355092945, 1355092941, 1355092938, 1355092935, 1355092934, 1355092920, 1355092914, 1355092911, 1355092909, 1355092905, 1355092904, 1355092887, 1355092884, 1355092881, 1355092880, 1355092877, 1355092876, 1355092858, 1355092855, 1355092852, 1355092850, 1355092847, 1355092846, 1355092828, 1355092823, 1355092816, 1355092812, 1355092809, 1355092808, 1355092738, 1355092728, 1355092713, 1355092711, 1355092706, 1355092704, 1355092687, 1355092683, 1355092678, 1355092674, 1355092672, 1355092670, 1355092653, 1355092649, 1355092644, 1355092641, 1355092637, 1355092635, 1355092573, 1355092570, 1355092567, 1355092564, 1355092559, 1355092557, 1355092515, 1355092511, 1355092507, 1355092503, 1355092500, 1355092498, 1355092484, 1355092481, 1355092477, 1355092476, 1355092473, 1355092471, 1355092454, 1355092452] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.08847713470458984 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 138.23553252220154 time spend to save output : 0.09342098236083984 total time spend for step 0 : 138.32895350456238 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 12.97user 3.59system 2:50.89elapsed 9%CPU (0avgtext+0avgdata 1753292maxresident)k 568136inputs+365152outputs (882major+464843minor)pagefaults 0swaps