python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/datou.py -j batch_current -C 2930218' -s rattrapage -M 0 -S 0 -U 100,80,95 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/home/admin/workarea/git/apy', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3602885 load datou : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 0 init_dummy_multi_datou is not linked in the step_by_step architecture ! WARNING : step 1294 init_dummy_multi_datou is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 3669>1000 please execute split_portfolio.py -i 23235267 -l 1000 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 4892, datou_cur_ids : ['2930218'] with mtr_portfolio_ids : ['23235267'] and first list_photo_ids : [] new path : /proc/3602885/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax over limit max, limiting to limit_max 300 list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 300 ; length of list_pids : 300 ; length of list_args : 300 time to download the photos : 60.28226613998413 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Mon May 26 12:41:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 2 we are using the classfication for multi_thcl [3513, 3892] begin to check gpu status inside check gpu memory 2025-05-26 12:41:54.680674: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-26 12:41:54.711092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 12:41:54.711424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 12:41:54.714218: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 12:41:54.716555: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 12:41:54.717095: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 12:41:54.719748: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 12:41:54.721535: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 12:41:54.726682: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 12:41:54.727958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 12:41:54.728435: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-26 12:41:54.759098: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-26 12:41:54.761085: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7e14000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-26 12:41:54.761139: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-26 12:41:54.770850: W tensorflow/compiler/xla/service/platform_util.cc:210] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory; total memory reported: 11553341440 2025-05-26 12:41:54.771004: I tensorflow/compiler/jit/xla_gpu_device.cc:161] Ignoring visible XLA_GPU_JIT device. Device number is 0, reason: Internal: no supported devices found for platform CUDA 2025-05-26 12:41:54.771838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 12:41:54.771927: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 12:41:54.771952: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 12:41:54.771975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 12:41:54.771997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 12:41:54.772019: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 12:41:54.772041: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 12:41:54.772077: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 12:41:54.773042: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 12:41:54.773102: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 l 3637 free memory gpu now : 3232 max_wait_temp : 1 max_wait : 5 ERROR in datou_step_exec, will save and exit ! CUDA runtime implicit initialization on GPU:0 failed. Status: out of memory File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3074, in datou_step_tfhub2 logical_gpus = tf.config.experimental.list_logical_devices('GPU') File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/config.py", line 372, in list_logical_devices return context.context().list_logical_devices(device_type=device_type) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1251, in list_logical_devices self.ensure_initialized() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 515, in ensure_initialized context_handle = pywrap_tfe.TFE_NewContext(opts) [1360597921, 1360597839, 1360597802, 1360597771, 1360597765, 1360597756, 1360597731, 1360597578, 1360597545, 1360597539, 1360597534, 1360597527, 1360597523, 1360597495, 1360597488, 1360597484, 1360597481, 1360597478, 1360597476, 1360597468, 1360597464, 1360597460, 1360597456, 1360597451, 1360597448, 1360597441, 1360597438, 1360597427, 1360597402, 1360597375, 1360597341, 1360597266, 1360597246, 1360597232, 1360597203, 1360597159, 1360597134, 1360597068, 1360597042, 1360597014, 1360596986, 1360596975, 1360596968, 1360596887, 1360596884, 1360596881, 1360596858, 1360596827, 1360596793, 1360596735, 1360596725, 1360596723, 1360596704, 1360596682, 1360596657, 1360596566, 1360596531, 1360596495, 1360596443, 1360596388, 1360596356, 1360596177, 1360596135, 1360596089, 1360596049, 1360596018, 1360596003, 1360595988, 1360595983, 1360595979, 1360595974, 1360595968, 1360595962, 1360595935, 1360595927, 1360595920, 1360595915, 1360595909, 1360595900, 1360595710, 1360595667, 1360595641, 1360595618, 1360595576, 1360595542, 1360595497, 1360595459, 1360595415, 1360595368, 1360595322, 1360595271, 1360594999, 1360594980, 1360594955, 1360594895, 1360594862, 1360594819, 1360594559, 1360594539, 1360594486, 1360594434, 1360594384, 1360594324, 1360594005, 1360593958, 1360593908, 1360593857, 1360593813, 1360593776, 1360593682, 1360593677, 1360593671, 1360593667, 1360593661, 1360593658, 1360593635, 1360593629, 1360593624, 1360593621, 1360593619, 1360593615, 1360593594, 1360593584, 1360593567, 1360593560, 1360593556, 1360593551, 1360593532, 1360593509, 1360593476, 1360593445, 1360593415, 1360593410, 1360593399, 1360593363, 1360593327, 1360593301, 1360593275, 1360593248, 1360593197, 1360593183, 1360593171, 1360593161, 1360593142, 1360593116, 1360593050, 1360593027, 1360592998, 1360592971, 1360592943, 1360592917, 1360592671, 1360592617, 1360592559, 1360592513, 1360592460, 1360592387, 1360592201, 1360592127, 1360592055, 1360592003, 1360591947, 1360591882, 1360591748, 1360591705, 1360591652, 1360591627, 1360591603, 1360591576, 1360591480, 1360591456, 1360591433, 1360591409, 1360591383, 1360591358, 1360591286, 1360591257, 1360591222, 1360591183, 1360591132, 1360591076, 1360590838, 1360590812, 1360590779, 1360590752, 1360590725, 1360590714, 1360590629, 1360590579, 1360590530, 1360590483, 1360590446, 1360590415, 1360590368, 1360590350, 1360590342, 1360590340, 1360590338, 1360590333, 1360590243, 1360590194, 1360590146, 1360590123, 1360590090, 1360590040, 1360589966, 1360589919, 1360589837, 1360589754, 1360589686, 1360589641, 1360589420, 1360589375, 1360589335, 1360589302, 1360589276, 1360589229, 1360588970, 1360588944, 1360588900, 1360588866, 1360588862, 1360588857, 1360588839, 1360588833, 1360588828, 1360588820, 1360588813, 1360588804, 1360588782, 1360588777, 1360588772, 1360588767, 1360588760, 1360588756, 1360588727, 1360588724, 1360588721, 1360588716, 1360588710, 1360588707, 1360588687, 1360588682, 1360588676, 1360588670, 1360588653, 1360588637, 1360588510, 1360588507, 1360588502, 1360588495, 1360588485, 1360588477, 1360588464, 1360588460, 1360588456, 1360588452, 1360588437, 1360588415, 1360588296, 1360588278, 1360588251, 1360588244, 1360588230, 1360588182, 1360587924, 1360587914, 1360587899, 1360587880, 1360587863, 1360587839, 1360587749, 1360587717, 1360587689, 1360587672, 1360587668, 1360587663, 1360587540, 1360587504, 1360587468, 1360587428, 1360587397, 1360587366, 1360587016, 1360586954, 1360586888, 1360586846, 1360586824, 1360586798, 1360586634, 1360586609, 1360586586, 1360586561, 1360586531, 1360586501, 1360586446, 1360586441, 1360586437, 1360586433, 1360586409] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 300 time used for this insertion : 0.12442159652709961 save_final ERROR in last step tfhub_classification2, CUDA runtime implicit initialization on GPU:0 failed. Status: out of memory time spend for datou_step_exec : 2.9759275913238525 time spend to save output : 0.13354134559631348 total time spend for step 0 : 3.109468936920166 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 4.45user 2.92system 1:06.52elapsed 11%CPU (0avgtext+0avgdata 438568maxresident)k 2144inputs+643552outputs (39major+148386minor)pagefaults 0swaps