python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 20 -a ' -a 4189' -s datou_current_4189 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 4027861 load datou : 4189 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 list_input_json: {} Current got : datou_id : 4189, datou_cur_ids : ['3646521'] with mtr_portfolio_ids : ['26397163'] and first list_photo_ids : [] new path : /proc/4027861/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : split_time_score over limit max, limiting to limit_max 100 list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB time to download the photos : 0.024377822875976562 About to test input to load Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 1 step1:split_time_score Mon Sep 1 15:00:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec begin split time score 2022-04-13 10:29:59 0 TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3379, 'mtr_user_id': 31, 'name': 'learn_classif_flux_maj_generique_effnet_v2_s_02062022', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'aluminium,ela,film_pedb,flux_dev,jrm,pcm,pcnc,pehd_pp,pet_clair,refus,tapis_vide', 'svm_portfolios_learning': '5515864,5515840,5515844,5515850,6244400,6237996,6237998,5515847,5515841,5515868,5515866', 'photo_hashtag_type': 4374, 'photo_desc_type': 5680, 'type_classification': 'tf_classification2', 'hashtag_id_list': '493546845,492741797,2107760237,2107760238,495916461,560181804,1284539308,2107760239,2107755846,538914404,2107748999'}] thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('14', 317),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {1: 311} 01092025 26397163 Nombre de photos uploadées : 317 / 23040 (1%) 01092025 26397163 Nombre de photos taguées (types de déchets): 311 / 317 (98%) 01092025 26397163 Nombre de photos taguées (volume) : 311 / 317 (98%) elapsed_time : load_data_split_time_score 4.5299530029296875e-06 elapsed_time : order_list_meta_photo_and_scores 0.00021839141845703125 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL?????? elapsed_time : fill_and_build_computed_from_old_data 0.017311573028564453 Catched exception ! Connect or reconnect ! Catched exception ! Connect or reconnect ! elapsed_time : insert_dashboard_record_day_entry 0.2384648323059082 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL Creating list_photo_by_hashtags Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None elapsed_time : list_photo_by_hashtags 0.03202319145202637 Creating list_photo_total elapsed_time : select_descriptors 5.738353252410889 01092025 26397163 Nombre de photos avec descriptors (type 5680) : 57 / 63 (90%) ERROR : Hum hum, what can we do for different size of descriptors (ignore the difference ) : 0 vs 1280 photo_id : 1380624987 photo_id_prec : 0 0:00:00|ON:LL0:00:00|OFF:LLLLLLLLLLLLLLLLLLLLLLLL0:02:00.038343|ON:LL0:01:00.615842|OFF:0:00:00|ON:LL0:00:00|OFF:LL0:00:59.836214|ON:LL0:01:59.750773|OFF:LL0:00:00|ON:LMissing descriptors for photos 1380627421 and 1380627469 LLMissing descriptors for photos 1380627590 and 1380627643 LMissing descriptors for photos 1380627643 and 1380628015 LMissing descriptors for photos 1380628015 and 1380628051 LMissing descriptors for photos 1380628051 and 1380628106 LLMissing descriptors for photos 1380628182 and 1380628223 LLMissing descriptors for photos 1380628480 and 1380628536 LMissing descriptors for photos 1380628536 and 1380628579 LMissing descriptors for photos 1380628579 and 1380628586 LMissing descriptors for photos 1380628586 and 1380628596 LMissing descriptors for photos 1380628596 and 1380628708 LMissing descriptors for photos 1380628708 and 1380628828 LMissing descriptors for photos 1380628828 and 1380628936 LMissing descriptors for photos 1380628936 and 1380628960 LMissing descriptors for photos 1380628960 and 1380628968 LMissing descriptors for photos 1380628968 and 1380629019 LMissing descriptors for photos 1380629019 and 1380629270 LMissing descriptors for photos 1380629270 and 1380629306 LMissing descriptors for photos 1380629306 and 1380629350 LMissing descriptors for photos 1380629350 and 1380629390 LMissing descriptors for photos 1380629390 and 1380629433 LMissing descriptors for photos 1380629433 and 1380629464 LMissing descriptors for photos 1380629464 and 1380629530 LMissing descriptors for photos 1380629530 and 1380629559 LMissing descriptors for photos 1380629559 and 1380629610 LMissing descriptors for photos 1380629610 and 1380629650 LMissing descriptors for photos 1380629650 and 1380629688 LMissing descriptors for photos 1380629688 and 1380629695 LMissing descriptors for photos 1380629695 and 1380630060 LMissing descriptors for photos 1380630060 and 1380630120 LMissing descriptors for photos 1380630120 and 1380630197 LMissing descriptors for photos 1380630197 and 1380630314 LMissing descriptors for photos 1380630314 and 1380630390 LMissing descriptors for photos 1380630390 and 1380630499 LMissing descriptors for photos 1380630499 and 1380630871 LMissing descriptors for photos 1380630871 and 1380630930 LMissing descriptors for photos 1380630930 and 1380630977 LMissing descriptors for photos 1380630977 and 1380631054 LMissing descriptors for photos 1380631054 and 1380631097 LMissing descriptors for photos 1380631097 and 1380631136 LMissing descriptors for photos 1380631136 and 1380631175 LMissing descriptors for photos 1380631175 and 1380631178 LMissing descriptors for photos 1380631178 and 1380631184 LMissing descriptors for photos 1380631184 and 1380631200 LMissing descriptors for photos 1380631200 and 1380631208 LMissing descriptors for photos 1380631208 and 1380631211 LMissing descriptors for photos 1380631211 and 1380631296 LMissing descriptors for photos 1380631296 and 1380631337 LMissing descriptors for photos 1380631337 and 1380631384 LMissing descriptors for photos 1380631384 and 1380631425 LMissing descriptors for photos 1380631425 and 1380631469 LMissing descriptors for photos 1380631469 and 1380631512 LMissing descriptors for photos 1380631512 and 1380631623 LMissing descriptors for photos 1380631623 and 1380631655 LMissing descriptors for photos 1380631655 and 1380631660 LMissing descriptors for photos 1380631660 and 1380631671 LMissing descriptors for photos 1380631671 and 1380631674 LMissing descriptors for photos 1380631674 and 1380631693 LMissing descriptors for photos 1380631693 and 1380631723 LMissing descriptors for photos 1380631723 and 1380631726 LMissing descriptors for photos 1380631726 and 1380631728 LMissing descriptors for photos 1380631728 and 1380631729 LMissing descriptors for photos 1380631729 and 1380631730 LMissing descriptors for photos 1380631730 and 1380631731 LMissing descriptors for photos 1380631731 and 1380631754 LMissing descriptors for photos 1380631754 and 1380631756 LMissing descriptors for photos 1380631756 and 1380631757 LMissing descriptors for photos 1380631757 and 1380631758 LMissing descriptors for photos 1380631758 and 1380631763 LMissing descriptors for photos 1380631763 and 1380631764 LMissing descriptors for photos 1380631764 and 1380631794 LMissing descriptors for photos 1380631794 and 1380631799 LMissing descriptors for photos 1380631799 and 1380631803 LMissing descriptors for photos 1380631803 and 1380631806 LMissing descriptors for photos 1380631806 and 1380631809 LMissing descriptors for photos 1380631809 and 1380631811 LMissing descriptors for photos 1380631811 and 1380631990 LMissing descriptors for photos 1380631990 and 1380632003 LMissing descriptors for photos 1380632003 and 1380632006 LMissing descriptors for photos 1380632006 and 1380632031 LMissing descriptors for photos 1380632031 and 1380632044 LMissing descriptors for photos 1380632044 and 1380632050 LMissing descriptors for photos 1380632050 and 1380632083 LMissing descriptors for photos 1380632083 and 1380632086 LMissing descriptors for photos 1380632086 and 1380632089 LMissing descriptors for photos 1380632089 and 1380632090 LMissing descriptors for photos 1380632090 and 1380632091 LMissing descriptors for photos 1380632091 and 1380632092 LMissing descriptors for photos 1380632092 and 1380632117 LMissing descriptors for photos 1380632117 and 1380632129 LMissing descriptors for photos 1380632129 and 1380632133 LMissing descriptors for photos 1380632133 and 1380632143 LMissing descriptors for photos 1380632143 and 1380632160 LMissing descriptors for photos 1380632160 and 1380632164 LMissing descriptors for photos 1380632164 and 1380632189 LMissing descriptors for photos 1380632189 and 1380632195 LMissing descriptors for photos 1380632195 and 1380632199 LMissing descriptors for photos 1380632199 and 1380632201 LMissing descriptors for photos 1380632201 and 1380632203 LMissing descriptors for photos 1380632203 and 1380632208 LMissing descriptors for photos 1380632208 and 1380632424 LMissing descriptors for photos 1380632424 and 1380632426 LMissing descriptors for photos 1380632426 and 1380632452 LMissing descriptors for photos 1380632452 and 1380632475 LMissing descriptors for photos 1380632475 and 1380632496 LMissing descriptors for photos 1380632496 and 1380632502 LMissing descriptors for photos 1380632502 and 1380632569 LMissing descriptors for photos 1380632569 and 1380632577 LMissing descriptors for photos 1380632577 and 1380632580 LMissing descriptors for photos 1380632580 and 1380632613 LMissing descriptors for photos 1380632613 and 1380632618 LMissing descriptors for photos 1380632618 and 1380632627 LMissing descriptors for photos 1380632627 and 1380632649 LMissing descriptors for photos 1380632649 and 1380632653 LMissing descriptors for photos 1380632653 and 1380632655 LMissing descriptors for photos 1380632655 and 1380632679 LMissing descriptors for photos 1380632679 and 1380632683 LMissing descriptors for photos 1380632683 and 1380632688 LMissing descriptors for photos 1380632688 and 1380632791 LMissing descriptors for photos 1380632791 and 1380632796 LMissing descriptors for photos 1380632796 and 1380632803 LMissing descriptors for photos 1380632803 and 1380632811 LMissing descriptors for photos 1380632811 and 1380632816 LMissing descriptors for photos 1380632816 and 1380632823 LMissing descriptors for photos 1380632823 and 1380632858 LMissing descriptors for photos 1380632858 and 1380632860 LMissing descriptors for photos 1380632860 and 1380632865 LMissing descriptors for photos 1380632865 and 1380632869 LMissing descriptors for photos 1380632869 and 1380632876 LMissing descriptors for photos 1380632876 and 1380632889 LMissing descriptors for photos 1380632889 and 1380632927 LMissing descriptors for photos 1380632927 and 1380632931 LMissing descriptors for photos 1380632931 and 1380632933 LMissing descriptors for photos 1380632933 and 1380632934 LMissing descriptors for photos 1380632934 and 1380632935 LMissing descriptors for photos 1380632935 and 1380632936 LMissing descriptors for photos 1380632936 and 1380633007 LMissing descriptors for photos 1380633007 and 1380633013 LMissing descriptors for photos 1380633013 and 1380633017 LMissing descriptors for photos 1380633017 and 1380633020 LMissing descriptors for photos 1380633020 and 1380633028 LMissing descriptors for photos 1380633028 and 1380633033 LMissing descriptors for photos 1380633033 and 1380633141 LMissing descriptors for photos 1380633141 and 1380633159 LMissing descriptors for photos 1380633159 and 1380633170 LMissing descriptors for photos 1380633170 and 1380633179 LMissing descriptors for photos 1380633179 and 1380633199 LMissing descriptors for photos 1380633199 and 1380633202 LMissing descriptors for photos 1380633202 and 1380633235 LMissing descriptors for photos 1380633235 and 1380633241 LMissing descriptors for photos 1380633241 and 1380633244 LMissing descriptors for photos 1380633244 and 1380633258 LMissing descriptors for photos 1380633258 and 1380633293 LMissing descriptors for photos 1380633293 and 1380633331 LMissing descriptors for photos 1380633331 and 1380633607 LMissing descriptors for photos 1380633607 and 1380633609 LMissing descriptors for photos 1380633609 and 1380633613 LMissing descriptors for photos 1380633613 and 1380633624 LMissing descriptors for photos 1380633624 and 1380633629 LMissing descriptors for photos 1380633629 and 1380633635 LMissing descriptors for photos 1380633635 and 1380633660 LMissing descriptors for photos 1380633660 and 1380633691 LMissing descriptors for photos 1380633691 and 1380633696 LMissing descriptors for photos 1380633696 and 1380633697 LMissing descriptors for photos 1380633697 and 1380633699 LMissing descriptors for photos 1380633699 and 1380633705 LMissing descriptors for photos 1380633705 and 1380633728 LMissing descriptors for photos 1380633728 and 1380633729 LMissing descriptors for photos 1380633729 and 1380633734 LMissing descriptors for photos 1380633734 and 1380633735 LMissing descriptors for photos 1380633735 and 1380633738 LMissing descriptors for photos 1380633738 and 1380633740 LMissing descriptors for photos 1380633740 and 1380633752 LMissing descriptors for photos 1380633752 and 1380633753 LMissing descriptors for photos 1380633753 and 1380633754 LMissing descriptors for photos 1380633754 and 1380633768 LMissing descriptors for photos 1380633768 and 1380633775 LMissing descriptors for photos 1380633775 and 1380633779 LMissing descriptors for photos 1380633779 and 1380633806 LMissing descriptors for photos 1380633806 and 1380633807 LMissing descriptors for photos 1380633807 and 1380633808 LMissing descriptors for photos 1380633808 and 1380633812 LMissing descriptors for photos 1380633812 and 1380633814 LMissing descriptors for photos 1380633814 and 1380633817 LMissing descriptors for photos 1380633817 and 1380633835 LMissing descriptors for photos 1380633835 and 1380633838 LMissing descriptors for photos 1380633838 and 1380633840 LMissing descriptors for photos 1380633840 and 1380633841 LMissing descriptors for photos 1380633841 and 1380633855 LMissing descriptors for photos 1380633855 and 1380633869 LMissing descriptors for photos 1380633869 and 1380633909 LMissing descriptors for photos 1380633909 and 1380633921 LMissing descriptors for photos 1380633921 and 1380633922 LMissing descriptors for photos 1380633922 and 1380633925 LMissing descriptors for photos 1380633925 and 1380633928 LMissing descriptors for photos 1380633928 and 1380633960 LMissing descriptors for photos 1380633960 and 1380634033 LMissing descriptors for photos 1380634033 and 1380634048 LMissing descriptors for photos 1380634048 and 1380634061 LMissing descriptors for photos 1380634061 and 1380634068 LMissing descriptors for photos 1380634068 and 1380634076 LMissing descriptors for photos 1380634076 and 1380634099 LMissing descriptors for photos 1380634099 and 1380634152 LMissing descriptors for photos 1380634152 and 1380634189 LMissing descriptors for photos 1380634189 and 1380634199 LMissing descriptors for photos 1380634199 and 1380634202 LMissing descriptors for photos 1380634202 and 1380634208 LMissing descriptors for photos 1380634208 and 1380634236 LMissing descriptors for photos 1380634236 and 1380634334 LMissing descriptors for photos 1380634334 and 1380634336 LMissing descriptors for photos 1380634336 and 1380634337 LMissing descriptors for photos 1380634337 and 1380634341 LMissing descriptors for photos 1380634341 and 1380634346 LMissing descriptors for photos 1380634346 and 1380634352 LMissing descriptors for photos 1380634352 and 1380634454 LMissing descriptors for photos 1380634454 and 1380634463 LMissing descriptors for photos 1380634463 and 1380634470 LMissing descriptors for photos 1380634470 and 1380634478 LMissing descriptors for photos 1380634478 and 1380634479 LMissing descriptors for photos 1380634479 and 1380634484 LMissing descriptors for photos 1380634484 and 1380634563 LMissing descriptors for photos 1380634563 and 1380634568 LMissing descriptors for photos 1380634568 and 1380634570 LMissing descriptors for photos 1380634570 and 1380634585 LMissing descriptors for photos 1380634585 and 1380634586 LMissing descriptors for photos 1380634586 and 1380634609 LMissing descriptors for photos 1380634609 and 1380634623 LMissing descriptors for photos 1380634623 and 1380634629 LMissing descriptors for photos 1380634629 and 1380634635 LMissing descriptors for photos 1380634635 and 1380634641 LMissing descriptors for photos 1380634641 and 1380634647 LMissing descriptors for photos 1380634647 and 1380634652 LMissing descriptors for photos 1380634652 and 1380634668 LMissing descriptors for photos 1380634668 and 1380634669 LMissing descriptors for photos 1380634669 and 1380634670 LMissing descriptors for photos 1380634670 and 1380634671 LMissing descriptors for photos 1380634671 and 1380634691 LMissing descriptors for photos 1380634691 and 1380634695 LMissing descriptors for photos 1380634695 and 1380634731 LMissing descriptors for photos 1380634731 and 1380634732 LMissing descriptors for photos 1380634732 and 1380634733 LMissing descriptors for photos 1380634733 and 1380634740 LMissing descriptors for photos 1380634740 and 1380634747 LMissing descriptors for photos 1380634747 and 1380634750 LMissing descriptors for photos 1380634750 and 1380634764 LMissing descriptors for photos 1380634764 and 1380634765 LMissing descriptors for photos 1380634765 and 1380634778 LMissing descriptors for photos 1380634778 and 1380634816 LMissing descriptors for photos 1380634816 and 1380634856 LMissing descriptors for photos 1380634856 and 1380634894 LMissing descriptors for photos 1380634894 and 1380635138 LMissing descriptors for photos 1380635138 and 1380635213 LMissing descriptors for photos 1380635213 and 1380635259 LMissing descriptors for photos 1380635259 and 1380635302 LMissing descriptors for photos 1380635302 and 1380635365 LMissing descriptors for photos 1380635365 and 1380635439 L 01092025 Removing 18 photos because of the 'same image' condition Total on : 180.366615 list_time_on Total off : 179.87455699999998 list_time_off dist_desc begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 317 time used for this insertion : 0.06734251976013184 photos_removed : len 18 elapsed_time : remove_photo_duplicate 0.10776400566101074 To do, maybe not at the correct place ! ..L.L.L.L.L.L.L.L.L.L.L.L.L.....L.....L.L............L.L.................L.L..L.L.L.L.L..L.L..L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.Lforce hashtag to JRM elapsed_time : CREATE_PORT_BATCH_BY_HOUR 0.0160369873046875 NUMBER BATCH : 2 list_ponderation used : [1e-05, 1e-05, 1e-05, 1e-05, 1e-05] , list_hashtag_class_create_as_list : ['jrm'] result_one_balle_Type_JRM:{'day': '01092025', 'map_nb_amount': {0: 6, 1: 33, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 600.3474371433258, 'nb_balles_papier': 0.00039000000000000037, 'begin_time_port': 'image_01092025_14_00_02_240542m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.00039000000000000037 We have rejected 6 photos because of the batch_size condition ! NUMBER BATCH list_of_portfolios_to_create : 1 list_same_port_ids : [26398702] find same portfolio which already exist 26398702 , we will use it Qualite : 0.05920921478503715 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=26398702 AND mptpi.`type`=3726 To do elapsed_time : count_nb_balles_and_create_portfolio 0.12163853645324707 # DISPLAY ALL COLLECTED DATA : {'01092025': {'nb_upload': 317, 'nb_taggue_class': 311, 'nb_taggue_densite': 311, 'nb_descriptors': 57}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1380635439, 1380635365, 1380635302, 1380635259, 1380635213, 1380635138, 1380634894, 1380634856, 1380634816, 1380634778, 1380634765, 1380634764, 1380634750, 1380634747, 1380634740, 1380634733, 1380634732, 1380634731, 1380634695, 1380634691, 1380634671, 1380634670, 1380634669, 1380634668, 1380634652, 1380634647, 1380634641, 1380634635, 1380634629, 1380634623, 1380634609, 1380634586, 1380634585, 1380634570, 1380634568, 1380634563, 1380634484, 1380634479, 1380634478, 1380634470, 1380634463, 1380634454, 1380634352, 1380634346, 1380634341, 1380634337, 1380634336, 1380634334, 1380634236, 1380634208, 1380634202, 1380634199, 1380634189, 1380634152, 1380634099, 1380634076, 1380634068, 1380634061, 1380634048, 1380634033, 1380633960, 1380633928, 1380633925, 1380633922, 1380633921, 1380633909, 1380633869, 1380633855, 1380633841, 1380633840, 1380633838, 1380633835, 1380633817, 1380633814, 1380633812, 1380633808, 1380633807, 1380633806, 1380633779, 1380633775, 1380633768, 1380633754, 1380633753, 1380633752, 1380633740, 1380633738, 1380633735, 1380633734, 1380633729, 1380633728, 1380633705, 1380633699, 1380633697, 1380633696, 1380633691, 1380633660, 1380633635, 1380633629, 1380633624, 1380633613] Looping around the photos to save general results len do output : 1 /26397163Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380635439', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380635365', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380635302', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380635259', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380635213', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380635138', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634894', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634856', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634816', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634778', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634765', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634764', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634750', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634747', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634740', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634733', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634732', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634731', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634695', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634691', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634671', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634670', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634669', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634668', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634652', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634647', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634641', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634635', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634629', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634623', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634609', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634586', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634585', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634570', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634568', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634563', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634484', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634479', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634478', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634470', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634463', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634454', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634352', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634346', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634341', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634337', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634336', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634334', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634236', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634208', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634202', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634199', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634189', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634152', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634099', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634076', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634068', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634061', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634048', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380634033', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633960', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633928', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633925', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633922', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633921', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633909', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633869', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633855', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633841', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633840', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633838', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633835', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633817', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633814', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633812', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633808', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633807', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633806', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633779', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633775', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633768', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633754', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633753', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633752', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633740', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633738', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633735', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633734', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633729', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633728', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633705', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633699', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633697', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633696', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633691', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633660', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633635', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633629', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633624', None, None, None, None, None, '3646521') ('4189', None, None, None, None, None, None, None, '3646521') ('4189', '26397163', '1380633613', None, None, None, None, None, '3646521') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 101 time used for this insertion : 0.0389251708984375 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.382458209991455 time spend to save output : 0.0395503044128418 total time spend for step 1 : 6.422008514404297 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 1 set_done_treatment 1.79user 0.80system 0:09.88elapsed 26%CPU (0avgtext+0avgdata 105572maxresident)k 1000inputs+552outputs (1major+49561minor)pagefaults 0swaps