python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1419778 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 874>200 please execute split_portfolio.py -i 22083388 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2727226'] with mtr_portfolio_ids : ['22083388'] and first list_photo_ids : [] new path : /proc/1419778/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 174 ; length of list_pids : 174 ; length of list_args : 174 time to download the photos : 28.867992162704468 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Sun Apr 6 22:20:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1527 wait 20 seconds inside check gpu memory 2025-04-06 22:21:21.004815: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-04-06 22:21:21.035829: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-06 22:21:21.036092: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-06 22:21:21.038154: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-06 22:21:21.040176: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-06 22:21:21.040461: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-06 22:21:21.042205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-06 22:21:21.043160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-06 22:21:21.047005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-06 22:21:21.048262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-06 22:21:21.048586: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-04-06 22:21:21.075230: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-04-06 22:21:21.077165: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fd544000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-04-06 22:21:21.077208: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-04-06 22:21:21.286327: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xda39fc0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-04-06 22:21:21.286366: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-04-06 22:21:21.287567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-06 22:21:21.287651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-06 22:21:21.287681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-06 22:21:21.287708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-06 22:21:21.287734: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-06 22:21:21.287760: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-06 22:21:21.287791: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-06 22:21:21.287819: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-06 22:21:21.289561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-06 22:21:21.289628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-06 22:21:21.290611: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-04-06 22:21:21.290634: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-04-06 22:21:21.290647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-04-06 22:21:21.292229: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 6816 max_wait_temp : 2 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.27647066116333 time used to load_weights : 0.1704862117767334 0it [00:00, ?it/s]temp/1743970828_1419778_1350156558_0e9980a94d1194068b4e9196b1b14f68.jpg temp/1743970828_1419778_1350156425_d5d7b6bfc641e4a60b9fcfe7d013b4d7.jpg temp/1743970828_1419778_1350156421_5cc3a5d11eae8a2c499e4c3f8e2b2309.jpg temp/1743970828_1419778_1350156418_7a529b55fd3698395ef58c6897985a99.jpg temp/1743970828_1419778_1350156416_0bf68e73ca4c8873538d99727b02c9c7.jpg temp/1743970828_1419778_1350156414_64193d31790c092adf5388f8f97a95be.jpg temp/1743970828_1419778_1350156412_5aa01ad91fcc58ecb68fd179abb879ef.jpg temp/1743970828_1419778_1350156291_5a28566f72582e29dd826832d3192661.jpg temp/1743970828_1419778_1350156287_01f6dc245db4a8faf483164e9563ef3a.jpg temp/1743970828_1419778_1350156284_df2fa0dfd6f7448ec4e08f2627493d36.jpg temp/1743970828_1419778_1350156280_71398aa066cb86bdebeb2562b08f7bda.jpg temp/1743970828_1419778_1350156277_1e7afc293024471a3af5a18c324270fb.jpg temp/1743970828_1419778_1350156275_4c492423349f07b22cedc2219100a882.jpg temp/1743970828_1419778_1350156131_48aedf8bb1bf6ec0b96ec757ab965c4e.jpg temp/1743970828_1419778_1350156129_a0122b95a2822764ec443e87bca90acf.jpg temp/1743970828_1419778_1350156128_ea6718279af6256ebb453d37ec201aa0.jpg temp/1743970828_1419778_1350156126_b9fbcde6cfc17dba949ab4c4bd1f695d.jpg temp/1743970828_1419778_1350156123_5f168d945927edb356626a58befcdc98.jpg temp/1743970828_1419778_1350156119_a4ba71abd058dfa15d09154bf0258620.jpg temp/1743970828_1419778_1350155938_de1ac3bb2150e03c5cc5c63bad43c9dd.jpg temp/1743970828_1419778_1350155933_d4b102200369882d2321cd587619f28c.jpg temp/1743970828_1419778_1350155928_ccb8cf037426bbff5e09c0470f8e183e.jpg temp/1743970828_1419778_1350155924_62db995d5d5b5288473ded1bc065c797.jpg temp/1743970828_1419778_1350155921_a7f36211ae611846d6f4a10db6839469.jpg temp/1743970828_1419778_1350155918_e08151c4ba7083c5e0f5e025cba24e25.jpg temp/1743970828_1419778_1350155807_fdc55580f358e097fbda6232ce177ed3.jpg temp/1743970828_1419778_1350155803_242dd6114cfa81bca61d7c61ba7914b8.jpg temp/1743970828_1419778_1350155799_c045554e30dd646b13cfa4104052b29d.jpg temp/1743970828_1419778_1350155794_ec4a9687a78a6b39c557df687c83ada7.jpg temp/1743970828_1419778_1350155791_f4be4bd16d69d617b5301d85d225a9c6.jpg temp/1743970828_1419778_1350155788_e678def71521e24ceeb741c567a9d368.jpg temp/1743970828_1419778_1350155660_d35ea6ec70e29b3c4439dc85160b159f.jpg temp/1743970828_1419778_1350155658_a23762fd582e73cec5f1e152c1072503.jpg temp/1743970828_1419778_1350155656_a506f5b2526bb5726ca802b7f35cb1f5.jpg temp/1743970828_1419778_1350155654_cc030a5ac4bc394d23d3344256ec9c67.jpg temp/1743970828_1419778_1350155649_e40774cb2ff885a39972a9f0efcf6612.jpg temp/1743970828_1419778_1350155644_fe0a8fca71ceeee11485f5610190c423.jpg temp/1743970828_1419778_1350155260_061b41bbdb97c5ec64e62c6c8168c990.jpg temp/1743970828_1419778_1350155256_32bfaddcc67a5afdd7cded2f46416c1b.jpg temp/1743970828_1419778_1350155252_6eab5b03d27298b81d8070789ce6511e.jpg temp/1743970828_1419778_1350155248_e93d4199ce2c17e6a806ae7747e49628.jpg temp/1743970828_1419778_1350155242_6cf783f63cecaac1cae26bf41e014f74.jpg temp/1743970828_1419778_1350155240_87ed748482f52343eba74eeea9944ce5.jpg temp/1743970828_1419778_1350155124_0f792eb5ed4aaed9bee8be162a382528.jpg temp/1743970828_1419778_1350155120_8f0d5e3ec158d4f206be41c459e72b11.jpg temp/1743970828_1419778_1350155116_c0acb1fd4252df88232361e6139c5bb1.jpg temp/1743970828_1419778_1350155112_1863f107d6b2a7c3a1ec4464f82dc675.jpg temp/1743970828_1419778_1350155108_8ee1d577da4ff7cb3e9b3a27974fe05c.jpg temp/1743970828_1419778_1350155105_afe3b1d9d0bf13e928f33a7939157e54.jpg temp/1743970828_1419778_1350154978_21f76ce13123f305b0c58488bfe27eb6.jpg temp/1743970828_1419778_1350154974_83483553c34c837edf46893c0c6ee127.jpg temp/1743970828_1419778_1350154972_9c26209fbc157fae95ca2dfb8273737c.jpg temp/1743970828_1419778_1350154969_e61f1710bc8861bfea99fa38c9a280ce.jpg temp/1743970828_1419778_1350154965_eda589277888c2ab23cf9bf541dbe6e1.jpg temp/1743970828_1419778_1350154963_04ae9c4377d0964ad7207063397e5db7.jpg temp/1743970828_1419778_1350154841_ca969c5254d546eb9066f240c0821b8b.jpg temp/1743970828_1419778_1350154838_65826baa53a356d432bf907252414bab.jpg temp/1743970828_1419778_1350154833_3c4e35f1537bc2a7d3cacea008346a21.jpg temp/1743970828_1419778_1350154829_f91f93495a0394864a8cc52accb9a891.jpg temp/1743970828_1419778_1350154824_5027479e91abd480060172afc8198b60.jpg temp/1743970828_1419778_1350154819_d9fe1df7cffbaddd2fdb77721367cf9e.jpg temp/1743970828_1419778_1350154669_6967c290280bfeeb8ea88b11f55d04b0.jpg temp/1743970828_1419778_1350154665_964a4f14d58f012feee517886a65dbcd.jpg temp/1743970828_1419778_1350154660_802510ae624d36515bd3856624a37cab.jpg temp/1743970828_1419778_1350154656_c1d67ec785662a1943e9069bcff18c10.jpg temp/1743970828_1419778_1350154652_0726ffa304f62d849170f292623aa186.jpg temp/1743970828_1419778_1350154647_fde173cbdc0cea666e80b6bba1b0a266.jpg temp/1743970828_1419778_1350154642_78f24b8750f2e8076c679743a339bf0a.jpg temp/1743970828_1419778_1350154637_d3912bfd3b9cedf763253876cdb140d3.jpg temp/1743970828_1419778_1350154364_3fe20adf5086219e6885bb7e8df40cf4.jpg temp/1743970828_1419778_1350154362_fc4c5eed6dc86f5d9e1c1629a399f7d8.jpg temp/1743970828_1419778_1350154359_fa0060f706e43bc9754913e5f3030227.jpg temp/1743970828_1419778_1350154353_a94fdb4bea71a4ba66393ea8c80ff36a.jpg temp/1743970828_1419778_1350154350_a6e1d1698354986037cc2cf3ac3edd7d.jpg temp/1743970828_1419778_1350154344_993b74d733ebe01c27fc9dbb16eff713.jpg temp/1743970828_1419778_1350154017_c4579effb7fe35f332f1356d80fdbc88.jpg temp/1743970828_1419778_1350154013_c017b42468617694b5bdc43d3e3bf9c7.jpg temp/1743970828_1419778_1350154009_63d36d4b7c347bd681617b663d970f87.jpg temp/1743970828_1419778_1350154006_e70e10d083c1c33c7737abb6b50e8583.jpg temp/1743970828_1419778_1350154004_34f4bf0f4ff09686877c10cda6799a0f.jpg temp/1743970828_1419778_1350154000_67b01c3e8d3c6663b304cd9ba089989b.jpg temp/1743970828_1419778_1350153742_876d3ae7a124199361d75197ee329ed6.jpg temp/1743970828_1419778_1350153739_97e814fcfc94101a30a9924dd4773451.jpg temp/1743970828_1419778_1350153735_5f1ee0c8dde460ade2751774d6a9f69b.jpg temp/1743970828_1419778_1350153732_41a3f2e57f8036045d6f7e3030099d2a.jpg temp/1743970828_1419778_1350153729_af5eda67269ca03c88b7c294e6e5b4e4.jpg temp/1743970828_1419778_1350153726_ec1f4a4cd07a12a42ed43edfa98282e2.jpg temp/1743970828_1419778_1350153585_5ab9fe780fe117d909e69e0af5974938.jpg temp/1743970828_1419778_1350153582_f8695e95eb65994d818cf99d324dd728.jpg temp/1743970828_1419778_1350153507_8ef91fb2b20912263f3987643b589f18.jpg temp/1743970828_1419778_1350153500_b0fbc4f1a6164521ac71203d22e82527.jpg temp/1743970828_1419778_1350153350_a9f5d577954c6298e5b605e661284909.jpg temp/1743970828_1419778_1350153343_edb13c95dd59f19ac1451ffba017e354.jpg temp/1743970828_1419778_1350153287_a81fb50595395b0625793e9f17deb2b5.jpg temp/1743970828_1419778_1350153254_1e616724bf23baf28f8b2a2eb7c4476b.jpg temp/1743970828_1419778_1350153205_f20a6e235b2201956bce6c1212b8b5a7.jpg temp/1743970828_1419778_1350153145_4f381b794a19e63b772a893f9fb73b53.jpg temp/1743970828_1419778_1350152992_2d22675cfbc2138e5c30e405bcf8eb25.jpg temp/1743970828_1419778_1350152639_e829d0f03456b85f1c12a1f5ee564dd3.jpg temp/1743970828_1419778_1350152636_d689303aea6bf12d344fd2f887835b6f.jpg temp/1743970828_1419778_1350152574_101f8a579c65d9a229800be908567850.jpg temp/1743970828_1419778_1350152570_37ccaf4ee642e9a2f9a05e50bf2a2d09.jpg temp/1743970828_1419778_1350152568_174bb3ce3ab4ec9df8557a27407f979f.jpg temp/1743970828_1419778_1350152564_46a0f950a3dc26c4a2552228fcf1ca9f.jpg temp/1743970828_1419778_1350152560_e2de369f839e305f418308a72396a7f8.jpg temp/1743970828_1419778_1350152555_bebf4dcc1eaeea11a178eb313ae4bd3d.jpg temp/1743970828_1419778_1350152425_0accd6721ef201d500cd2641d167d2c3.jpg temp/1743970828_1419778_1350152421_9b00c2e622b61d48aec6bcd3315cd279.jpg temp/1743970828_1419778_1350152416_54bb2eeb22998491a283b5acd4a7478f.jpg temp/1743970828_1419778_1350152412_f3f71f7b9dc4109b5168849348f605d0.jpg temp/1743970828_1419778_1350152408_e0367899e7fba8f2973a19c15a5c5ff5.jpg temp/1743970828_1419778_1350152405_a6827f4d0db962c0d12e02fd6efb7734.jpg temp/1743970828_1419778_1350152221_c55607ae4a42eb0a7078d58aac7abded.jpg 174it [00:00, 2983.38it/s]2025-04-06 22:21:31.688878: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-06 22:21:31.904029: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-06 22:21:36.979900: E tensorflow/stream_executor/cuda/cuda_driver.cc:1045] failed to enqueue async memcpy from host to device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; GPU dst: 0x7fd3852fd400; host src: 0x7fd458294980; size: 602112=0x93000 2025-04-06 22:21:36.979999: E tensorflow/stream_executor/stream.cc:334] Error recording event in stream: Error recording CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; not marking stream as bad, as the Event object may be at fault. Monitor for further errors. 2025-04-06 22:21:36.980020: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-04-06 22:21:36.980032: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:273] Unexpected Event status: 1 Command terminated by signal 6 17.38user 4.46system 1:12.01elapsed 30%CPU (0avgtext+0avgdata 2703728maxresident)k 1736inputs+155784outputs (6major+701668minor)pagefaults 0swaps