python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/datou.py -j batch_current -C 2562432' -s traitement_4892 -M 0 -S 0 -U 100,80,95 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/home/admin/workarea/git/apy', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3317592 load datou : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 0 init_dummy_multi_datou is not linked in the step_by_step architecture ! WARNING : step 1294 init_dummy_multi_datou is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 7456>1000 please execute split_portfolio.py -i 20244820 -l 1000 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 4892, datou_cur_ids : ['2562432'] with mtr_portfolio_ids : ['20244820'] and first list_photo_ids : [] new path : /proc/3317592/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax over limit max, limiting to limit_max 300 list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 300 ; length of list_pids : 300 ; length of list_args : 300 time to download the photos : 61.08236336708069 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Fri Feb 7 11:50:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 2 we are using the classfication for multi_thcl [3513, 3890] begin to check gpu status inside check gpu memory 2025-02-07 11:50:34.489115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-07 11:50:34.526184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 11:50:34.528065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 11:50:34.557980: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 11:50:34.577315: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 11:50:34.581712: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 11:50:34.614161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 11:50:34.618703: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 11:50:34.669500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 11:50:34.743120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 11:50:34.743671: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-07 11:50:34.771352: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-07 11:50:34.772910: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7c94000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-07 11:50:34.772936: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-07 11:50:34.894450: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x267aa870 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-07 11:50:34.894501: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-07 11:50:34.895515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 11:50:34.895675: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 11:50:34.895718: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 11:50:34.895749: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 11:50:34.895781: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 11:50:34.895814: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 11:50:34.895846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 11:50:34.895876: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 11:50:34.897341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 11:50:34.897407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 11:50:34.898373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 11:50:34.898393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 11:50:34.898405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 11:50:34.900207: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.2653329372406 time used to load_weights : 0.20476865768432617 0it [00:00, ?it/s] 43it [00:00, 424.55it/s] 86it [00:00, 376.97it/s]temp/1738925369_3317592_1335003018_ca12b1227497cdb45cda11c4ad705bb6.jpg temp/1738925369_3317592_1335002829_274815d9f4958be4100decefc7406d4d.jpg temp/1738925369_3317592_1335002721_ad0d4549d4cf083b6ea9be1b88b0df8f.jpg temp/1738925369_3317592_1335002516_2b3c7f5271427bd88d817a762fd27235.jpg temp/1738925369_3317592_1335002511_b0589dc61a9059a8596087f2ac64b753.jpg temp/1738925369_3317592_1335002499_386ecf27f845682d55a7c54d0e3fb8f3.jpg temp/1738925369_3317592_1335002218_3a88c24d255187f0268828630e26c79d.jpg temp/1738925369_3317592_1335002209_a1f1cfdd68c5028dfe2a9befa68e3bf8.jpg temp/1738925369_3317592_1335002179_01ebf9bce99f79a513658faffc12fa9c.jpg temp/1738925369_3317592_1335002087_4627c6041eb3722418a0bffe886d04a7.jpg temp/1738925369_3317592_1335001980_4164102ed6a8aa8907c77dc9958f9b45.jpg temp/1738925369_3317592_1335001831_eb7c4009ac94e2a5197fc3b6cad2686e.jpg temp/1738925369_3317592_1335001185_34dddf732b7449945507b472d1f58725.jpg temp/1738925369_3317592_1335001184_27374ec3942bb0d29a5cb13cf5ec0963.jpg temp/1738925369_3317592_1335001182_cdf94b682d9257eaccb7c250134f662e.jpg temp/1738925369_3317592_1335001181_3ae950ecf4e7aa0935a956ff62d93fae.jpg temp/1738925369_3317592_1335001180_f13950570730dd373e4443a760c7aa1e.jpg temp/1738925369_3317592_1335001179_998db7eb283db71e8a7db0077afbedb2.jpg temp/1738925369_3317592_1335001040_6c858d8ebf6e2cc3cecf1ba375e68bce.jpg temp/1738925369_3317592_1335001037_13d3872ff474b1d81d17023018cee59e.jpg temp/1738925369_3317592_1335000926_60b6ed0b476cdb0d800fd133e31749d8.jpg temp/1738925369_3317592_1335000925_f3148ca0b6d78a6997d14d5d26740e03.jpg temp/1738925369_3317592_1335000921_1744a4451d2b055b295f16e0f009acfe.jpg temp/1738925369_3317592_1335000920_157f6bf803e289fd08d1629428ce8a53.jpg temp/1738925369_3317592_1335000894_7b3ee38309f4a9e1cd99fa3891e5b7ec.jpg temp/1738925369_3317592_1335000890_4112a46d710baf297b4d972564fb188b.jpg temp/1738925369_3317592_1335000886_d2322970db3ac006faf2c2e5d2c4e6cf.jpg temp/1738925369_3317592_1335000882_1f197438b13e117e14605c2c5cbb64ec.jpg temp/1738925369_3317592_1335000880_23fa292bbcba59bba9cb19011ae2cbf6.jpg temp/1738925369_3317592_1335000878_16275bfacb5a1223f56a0a12981c7b3a.jpg temp/1738925369_3317592_1335000872_7720e325d094111a4e80906165862743.jpg temp/1738925369_3317592_1335000868_7b79256b5d2a821556934e91c9b4dcf1.jpg temp/1738925369_3317592_1335000864_859d7f9fea602bf6ba4583af576cc88b.jpg temp/1738925369_3317592_1335000860_4cb8d234bb8d50d92ad186a59d9db259.jpg temp/1738925369_3317592_1335000855_00967b13c270662b783e955cbf6fd838.jpg temp/1738925369_3317592_1335000854_66f8e1c14f01ff85d4ca34b5448804df.jpg temp/1738925369_3317592_1335000830_49f0bf86ff3b5d9495d9e58c704bd993.jpg temp/1738925369_3317592_1335000826_aaae510eb870099c72f435c5dc0d4f56.jpg temp/1738925369_3317592_1335000824_207b8c7e73e4158da563b6f70b4fea07.jpg temp/1738925369_3317592_1335000815_ec8a214b9cc1f69a5e4df1890ff23908.jpg temp/1738925369_3317592_1335000808_ebe461e1e5351b374bfd50ef76610099.jpg temp/1738925369_3317592_1335000776_4561409d1b3a187490e7e60dabf25472.jpg temp/1738925369_3317592_1335000571_318a787b1972a487e1305d5adbf13abd.jpg temp/1738925369_3317592_1335000568_8bf9e10c65b49fc9be90c5a2542e0549.jpg temp/1738925369_3317592_1335000558_b0f511d96a0bdd77bd2bdc5773eb8dd8.jpg temp/1738925369_3317592_1335000551_fd3c6fc0b3350a88fde82275ca191504.jpg temp/1738925369_3317592_1335000495_abb8111c69ffeeb8a9cbc32bc07fd9f0.jpg temp/1738925369_3317592_1335000494_d3b3d0ee2d718016ca11110f0f9fcf60.jpg temp/1738925369_3317592_1335000478_2ac663df6d73c28715a10d646d1d9b33.jpg temp/1738925369_3317592_1335000476_129c8694c9556b344dc139ef810afb49.jpg temp/1738925369_3317592_1335000474_2b34e2477e38c68368e073c06cc1dc38.jpg temp/1738925369_3317592_1335000414_1a0ee71c80e3dd20bdadfba6fa16d288.jpg temp/1738925369_3317592_1335000323_8a91d5fd997f45ec65ce9fc8657a3764.jpg temp/1738925369_3317592_1335000240_c213bda39f2a90931d9308381ea6876f.jpg temp/1738925369_3317592_1334999928_637929610beeb6800b3a8d0be3da2b15.jpg temp/1738925369_3317592_1334999870_1f56fb254ed03aaccab04f9fd3e7bf48.jpg temp/1738925369_3317592_1334999804_c0c0b594cc5a982c96abe466efc037c0.jpg temp/1738925369_3317592_1334999731_ab6a873896914d1b318bdfbff279952a.jpg temp/1738925369_3317592_1334999665_370c45da426ae952b12c37fdba5fe991.jpg temp/1738925369_3317592_1334999658_9e4447029a0fb01d678a01ce48d849b7.jpg temp/1738925369_3317592_1334999648_dc12389424453d2aba3d01542c617490.jpg temp/1738925369_3317592_1334999639_e3f3759a53d6ab774778e4a17bf8f82f.jpg temp/1738925369_3317592_1334999630_1d2d50e243c0b849acbd44e78e263982.jpg temp/1738925369_3317592_1334999625_a7891c0c9dfdda326d0818c1ecd67396.jpg temp/1738925369_3317592_1334999621_bd1fc9c5740a5c022b812b894d0c770e.jpg temp/1738925369_3317592_1334999616_f7c68bc10b4df79079ff1d8123e99025.jpg temp/1738925369_3317592_1334999595_48894691d5920b7504b6c7b508509606.jpg temp/1738925369_3317592_1334999586_01420edfdd544af920269945f0aafda0.jpg temp/1738925369_3317592_1334999526_44b4380d643b8060e0823b6a796b9b37.jpg temp/1738925369_3317592_1334999512_dc75589f21aad454f564d7d89de12ec7.jpg temp/1738925369_3317592_1334999509_a77da346021a04fb20bad8831bf8c073.jpg temp/1738925369_3317592_1334999506_0ee2dfe5899abf4e7538b005c80d8e1a.jpg temp/1738925369_3317592_1334999375_067b975646a0cd6ab8a0f556a9c3e8ef.jpg temp/1738925369_3317592_1334999367_af2650ab0bddaf77b36b702ecdb89901.jpg temp/1738925369_3317592_1334999352_95c7f2a3b4b607a2bb58e48e9ba107ec.jpg temp/1738925369_3317592_1334999349_f101fcfbad8385013c02e422d364283b.jpg temp/1738925369_3317592_1334999347_16645eed0e761afac2b2f659f016131a.jpg temp/1738925369_3317592_1334999344_837549b591bc52910854bfecd64e7cd7.jpg temp/1738925369_3317592_1334999324_bce9735a5810d80b5a0dbf1573a8d8a7.jpg temp/1738925369_3317592_1334999317_b178849ccaa71f47c146d8e2366fdb1e.jpg temp/1738925369_3317592_1334999314_595580d975745c0d2e474237731d5dd9.jpg temp/1738925369_3317592_1334999311_b64b24d229c405b7c7217039e9b362e2.jpg temp/1738925369_3317592_1334999308_1563e2aa5d43c3c088327127b3e45b86.jpg temp/1738925369_3317592_1334999305_bcd4af526e8659e5234df0781ab2b741.jpg temp/1738925369_3317592_1334998976_51818cceb6b1085f570b424591f817f6.jpg temp/1738925369_3317592_1334998939_c6b5256e75a734903b796d2918fe6390.jpg temp/1738925369_3317592_1334998935_25b715b73dcc1211c8ee601ae347f339.jpg temp/1738925369_3317592_1334998927_d74c5110496f5d8a6f5b953d1db87d0d.jpg temp/1738925369_3317592_1334998913_1291a47c685a0653acdbddb8b114b0fd.jpg temp/1738925369_3317592_1334998896_1b059a55d88063ead202d811d47ef581.jpg temp/1738925369_3317592_1334998397_70b793f24b6ad84cdde6ad68cd2b7dc7.jpg temp/1738925369_3317592_1334998387_b8a4325f37216f18d05f59abdd15a5c0.jpg temp/1738925369_3317592_1334998370_fb43edd788bd4968c3a4701a5dce40da.jpg temp/1738925369_3317592_1334998286_8966ad7a7a9d0161f535310a7b86ee9b.jpg temp/1738925369_3317592_1334998257_97aab4ba969c2b41413d7f84c5766a59.jpg temp/1738925369_3317592_1334998252_1ddb856bf983cde14840bf62d5e973df.jpg temp/1738925369_3317592_1334998026_7ad6bcdc61a600fda6357e6af582c95d.jpg temp/1738925369_3317592_1334998022_2fe29e5ad3611b4ff7ab07b0f366ec69.jpg temp/1738925369_3317592_1334998017_9fe85f7d168975df72d79f4f5d85709d.jpg temp/1738925369_3317592_1334998013_079dc99fb7d0e121fd33298066cd8ef5.jpg temp/1738925369_3317592_1334998010_d07c6b8db379deac3cbf5878db7e4f5f.jpg temp/1738925369_3317592_1334998006_9a60efe038c38bafba5e310414f1adec.jpg temp/1738925369_3317592_1334997983_9a906861260f634faa50673c233646b0.jpg temp/1738925369_3317592_1334997979_6530e65e95eb5b5857d43b8b689b4609.jpg temp/1738925369_3317592_1334997975_6244cd43b839d754f2e0b0e9c7d4bac2.jpg temp/1738925369_3317592_1334997973_737ebfc7bcd1b13e7c0344e9d6f5ebae.jpg temp/1738925369_3317592_1334997972_ad61060976c3918d89cd5920c389a496.jpg temp/1738925369_3317592_1334997969_cb893604f7668fdaf050111e51b7c3fc.jpg temp/1738925369_3317592_1334997930_27ac9f240824df2f696b27281ce31cf0.jpg temp/1738925369_3317592_1334997927_27d036e046e31063a5238ace9794bdf2.jpg temp/1738925369_3317592_1334997909_a033daee84c069793ecefe0d8d8869cf.jpg temp/1738925369_3317592_1334997896_5c825865ab80a6892f48fc4bab24648b.jpg temp/1738925369_3317592_1334997891_862a80ff7b1f9a2f02a7037990cc0170.jpg 160it [00:00, 529.62it/s]temp/1738925369_3317592_1334997888_dd2f1b300e4741e6ee8bba026442b0c2.jpg temp/1738925369_3317592_1334997858_73f42918bbb8f9d7f9e90a2a819137c2.jpg temp/1738925369_3317592_1334997855_d9e0bc3099710519292170f82158a711.jpg temp/1738925369_3317592_1334997852_27980942c07a1e4add0d7826c3c215b1.jpg temp/1738925369_3317592_1334997849_d5db7a21426d690fa7f879b52f29af74.jpg temp/1738925369_3317592_1334997845_806a44c0256cc26ee270288002163e98.jpg temp/1738925369_3317592_1334997841_574dabc025c408ff7e36e871a8368488.jpg temp/1738925369_3317592_1334997825_c6d32d0b32e593fa8bf3544548d621de.jpg temp/1738925369_3317592_1334997821_b7d38fcd7e6676ee3059edc6645cac21.jpg temp/1738925369_3317592_1334997817_c75bee3d3ff4572239bfae7a989c6c25.jpg temp/1738925369_3317592_1334997813_701fa80f6af1741b754f66852c7d491a.jpg temp/1738925369_3317592_1334997810_a6ec546459ccd93d6a07a27d8a12d7fb.jpg temp/1738925369_3317592_1334997807_f739e25e23f6a04889a5c1e41c075759.jpg temp/1738925369_3317592_1334997725_8f069405b40a8664c348ac420547f497.jpg temp/1738925369_3317592_1334997713_81d879ea7118147fbe918c2bf55da8f4.jpg temp/1738925369_3317592_1334997710_44b8d024e4bd64057b6921d901dbabd1.jpg temp/1738925369_3317592_1334997684_b96e0cb5295067e47a17e85215876c70.jpg temp/1738925369_3317592_1334997680_ecaed5f7de939c58921e8ca81106fde9.jpg temp/1738925369_3317592_1334997654_1eed9343eb7934280de271e9279de496.jpg temp/1738925369_3317592_1334997542_a94e11ed1f4509fa84676ba4a3bee07c.jpg temp/1738925369_3317592_1334997538_8aab5f1b9865049557cf160514d0060f.jpg temp/1738925369_3317592_1334997535_a75ebf3bac38a32c09638da62f71d490.jpg temp/1738925369_3317592_1334997531_41af551a00dcc511f5d92dd37c91e1ac.jpg temp/1738925369_3317592_1334997528_41bbbb83150e3cdaea53f87e96bac485.jpg temp/1738925369_3317592_1334997525_37050b80ce3131963be7730fa2fce27d.jpg temp/1738925369_3317592_1334997445_52d281bd720da899e0f709e22f471c5f.jpg temp/1738925369_3317592_1334997442_7d2631a9f348a38f9111dad7a3807b54.jpg temp/1738925369_3317592_1334997426_81f5cb8881eff3f65de312adc17e9a8c.jpg temp/1738925369_3317592_1334997419_fd38e56073fbf515957cfbd4b87303a8.jpg temp/1738925369_3317592_1334997414_e1ca23c89695afa8ecacf81d15777335.jpg temp/1738925369_3317592_1334997367_2754ceae4071ba21bf09212df667547a.jpg temp/1738925369_3317592_1334997351_619d699dc27b5e361112ce798ba78c6d.jpg temp/1738925369_3317592_1334997321_ef66745da8e2256e2e79a710a08b31ce.jpg temp/1738925369_3317592_1334997314_37b20aeee440cc5c73fb95fbb082d83e.jpg temp/1738925369_3317592_1334997309_54ce8994b800565d227f9a3d898be099.jpg temp/1738925369_3317592_1334997247_b1f375fd8fa1821f1a3bcefa12725a28.jpg temp/1738925369_3317592_1334997075_98165216c8d1ed003b113388d50a2ea9.jpg temp/1738925369_3317592_1334996641_9aad69776f931ef9f81553bfa8913e61.jpg temp/1738925369_3317592_1334996397_02a860b60181e168279aab5cfd4337ed.jpg temp/1738925369_3317592_1334996173_46a4116e7b9e9f14674e54da33b3c776.jpg temp/1738925369_3317592_1334996123_dec40461b0788a8ff8f11fd19cccb8cb.jpg temp/1738925369_3317592_1334996118_163b8520526c21aacda37ec359e619c0.jpg temp/1738925369_3317592_1334996115_b9c77c8e4a909c96c305ab193f63f8e7.jpg temp/1738925369_3317592_1334995470_0e7fe9f14e7eb2b8ab7fb0b2f5c0a541.jpg temp/1738925369_3317592_1334995464_1cc6f5a7ee2f63d46c5420dd7172b434.jpg temp/1738925369_3317592_1334995459_20071265b3e27345e62c704095490d73.jpg temp/1738925369_3317592_1334995456_5953a6f89d04e583cae7e7df4abea390.jpg temp/1738925369_3317592_1334995454_d8a5b5fd6cd3a2cf492bc1dfff0f06e7.jpg temp/1738925369_3317592_1334995451_4c9675205106dc240f04392b706f8c83.jpg temp/1738925369_3317592_1334995347_7283a548c38faa2c617883c89524421e.jpg temp/1738925369_3317592_1334995343_1267d18c8253920331e92bb1892c4461.jpg temp/1738925369_3317592_1334995340_8d9281374a8eff909bc7d39f4a106c84.jpg temp/1738925369_3317592_1334995338_fcded11ae96ff676fa837351f1ce419d.jpg temp/1738925369_3317592_1334995336_14005dd678703c68d79956d5e3e84968.jpg temp/1738925369_3317592_1334995334_09eaf5995a10b7766143311069de1c0c.jpg temp/1738925369_3317592_1334995168_4c3358f001c01e2c8555f19b6e4d3564.jpg temp/1738925369_3317592_1334995166_daf97948988ef26f03de1ffbb183298b.jpg temp/1738925369_3317592_1334995165_f0199247e7816a98b3305ce84427e655.jpg temp/1738925369_3317592_1334995162_7dfc2ed6cccb0c47b377dca13425493d.jpg temp/1738925369_3317592_1334995158_acaede3abf87a008cb058b3bc604ceea.jpg temp/1738925369_3317592_1334995141_4c7e8486cf030b9adb2eb0b77b05a9af.jpg temp/1738925369_3317592_1334995102_a4f15620ed0e57099fd83ba535638510.jpg temp/1738925369_3317592_1334995099_15d22d2bf16fc93e1c29befbeafe24d9.jpg temp/1738925369_3317592_1334995096_8e0bcf39d8b8d1400dac5a57b660bd12.jpg temp/1738925369_3317592_1334995070_46edd9715296a46b88febaed444478e9.jpg temp/1738925369_3317592_1334995035_6fbc1b4873059f49e429607f5c7060df.jpg temp/1738925369_3317592_1334995027_2faa728da4498d1293ddeba03844b5d6.jpg temp/1738925369_3317592_1334994785_823062a8e931dcdc788e461942897b28.jpg temp/1738925369_3317592_1334994723_f478679f1fcedca67fe6890c5002d92c.jpg temp/1738925369_3317592_1334994718_cba0846f94744377edd0add5b570f8e2.jpg temp/1738925369_3317592_1334994715_ec0397ec8dbc97484f547883d16e9b48.jpg temp/1738925369_3317592_1334994705_d0b245677542315293c418563a29c12b.jpg temp/1738925369_3317592_1334994700_322b959b7d1fb24370b1c9798c177666.jpg temp/1738925369_3317592_1334994668_be834b0ae15374275c3d7223fab2f19d.jpg temp/1738925369_3317592_1334994650_03d0ae04ed299f37605e779a0c7b99d3.jpg temp/1738925369_3317592_1334994649_b75642725a348ab70462eadbc7102fea.jpg temp/1738925369_3317592_1334994645_988493ac9c56cead331ee6faccde710e.jpg temp/1738925369_3317592_1334994644_43d655c324306bef65c51309254f5577.jpg temp/1738925369_3317592_1334994640_1af257932f23393a559b1ab09f2aa3da.jpg temp/1738925369_3317592_1334994638_1879d95efb6ae559444a8538a5f280cf.jpg temp/1738925369_3317592_1334994630_fbd56be275cfd20362648ffa90442cbd.jpg temp/1738925369_3317592_1334994629_06296fd8a32996db6f9f0c257e65857a.jpg temp/1738925369_3317592_1334994621_fc8bd7680e5d874d4e5bbd24ec3f4e9d.jpg temp/1738925369_3317592_1334994620_bd05ef00f689f0d08f96d097276db013.jpg temp/1738925369_3317592_1334994615_a94b7c3e48569ab1ca21078dccc2967e.jpg temp/1738925369_3317592_1334994585_5e0cbffe194489007b3de4d1bf1c581a.jpg temp/1738925369_3317592_1334994582_30125c2655e7b7019413762a8e0bafb2.jpg temp/1738925369_3317592_1334994579_7b7de40dec5dd3c960ca3edd2946af13.jpg temp/1738925369_3317592_1334994574_7b235d755f7eff481c7eefe517fafbd6.jpg temp/1738925369_3317592_1334994570_d12d6a18e43979b97c9599a528913d0d.jpg temp/1738925369_3317592_1334994565_28c2d13a2af9cd130156044642296471.jpg temp/1738925369_3317592_1334994541_455fc06819982f7873199529be65ff1e.jpg temp/1738925369_3317592_1334994532_2c06c2929aabb4103b98eacf47e713cf.jpg temp/1738925369_3317592_1334994529_562072d73ae3ea903e6ab56f19c59f55.jpg temp/1738925369_3317592_1334994525_0fd5c3fc55313d0fd0f6191571e674f1.jpg temp/1738925369_3317592_1334994523_d066cb4feebe8c1323a3a98413e850dd.jpg temp/1738925369_3317592_1334994519_bc530e5b9ecfa92e04a540ca0a130d50.jpg temp/1738925369_3317592_1334994387_3a59dede543814f0812f70aa88dfed36.jpg temp/1738925369_3317592_1334994377_9c6bec79f52b57808c7e6b666bc4c0e2.jpg temp/1738925369_3317592_1334994312_7c488a1cfd6540865e754965bdc2c5ea.jpg temp/1738925369_3317592_1334994220_e6a12100e94aa89a046b6fffad8c2bff.jpg temp/1738925369_3317592_1334994218_1442d776bb97d97d0bc2dd388faf8753.jpg temp/1738925369_3317592_1334994217_3fa4f9c93e33f6a630451cc8750f46b3.jpg temp/1738925369_3317592_1334994161_b9d7aeab2b5b95d4eb7ea6c2654f167b.jpg temp/1738925369_3317592_1334994159_51c0264bcb77606b76050d4ca877024b.jpg temp/1738925369_3317592_1334994158_e0b783e860a8499f8032028a65a44384.jpg temp/1738925369_3317592_1334994155_78b536b7d239b2846af67ce5079d60d3.jpg temp/1738925369_3317592_1334994153_b27dc44a9b85dae568b44f3410e704d1.jpg temp/1738925369_3317592_1334994150_0e33d88d86ab0adeb5a8929d31c493dd.jpg temp/1738925369_3317592_1334994148_01fde98f57fce175b678eb25bd6faec3.jpg temp/1738925369_3317592_1334994146_d92dee395c8f6df283bf637b0303f7ee.jpg temp/1738925369_3317592_1334994144_25bdc1c407b14ad3eae8b0b303c3f64d.jpg temp/1738925369_3317592_1334994142_d49d460a29cfff84c03c40b08afa5b3b.jpg 251it [00:00, 671.82it/s] 300it [00:00, 605.75it/s]2025-02-07 11:50:46.348260: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 11:50:46.855030: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 11:51:01.345646: E tensorflow/stream_executor/cuda/cuda_driver.cc:1045] failed to enqueue async memcpy from host to device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; GPU dst: 0x7f7ad12fd400; host src: 0x7f7b98001440; size: 602112=0x93000 2025-02-07 11:51:01.345803: E tensorflow/stream_executor/stream.cc:334] Error recording event in stream: Error recording CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; not marking stream as bad, as the Event object may be at fault. Monitor for further errors. 2025-02-07 11:51:01.345825: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-02-07 11:51:01.345839: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:273] Unexpected Event status: 1 Command terminated by signal 6 28.09user 6.37system 1:33.85elapsed 36%CPU (0avgtext+0avgdata 2883820maxresident)k 1034560inputs+1098216outputs (2802major+802946minor)pagefaults 0swaps