Perf improvements with concurrent hash calculation ================================================= Using the source code dataset here are the new perfs: ``` 254 541,85 msec task-clock # 1,333 CPUs utilized 489 390 context-switches # 0,002 M/sec 14 491 cpu-migrations # 0,057 K/sec 109 170 page-faults # 0,429 K/sec 702 598 342 141 cycles # 2,760 GHz 1 191 229 091 705 instructions # 1,70 insn per cycle 172 579 644 365 branches # 678,001 M/sec 2 502 920 412 branch-misses # 1,45% of all branches 191,024430360 seconds time elapsed 247,992304000 seconds user 15,759037000 seconds sys ``` But this time I also had the good idea to close all my processes and to use a tmp directory for writing. ------------------------------------------------------------------------------- With the same setup, the previous perf was: ``` 277 665,78 msec task-clock # 1,411 CPUs utilized 853 639 context-switches # 0,003 M/sec 27 276 cpu-migrations # 0,098 K/sec 110 187 page-faults # 0,397 K/sec 764 443 227 093 cycles # 2,753 GHz 1 221 696 199 089 instructions # 1,60 insn per cycle 178 891 873 274 branches # 644,271 M/sec 2 578 200 052 branch-misses # 1,44% of all branches 196,744991354 seconds time elapsed 270,030535000 seconds user 18,285378000 seconds sys ``` So not that big of an improvement, but it seems that at the same time CPU usage has decreased a bit. Maybe because less synchronization calls were made ? kup/log/testdata/repo_8k/00000/chunks/000000000000003'>
path: root/testdata/repo_8k/00000/chunks/000000000000003
blob: 8720f6bb309b47c8325ee18d4cab92dc8571a0a5 (plain) 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85

R] Temps pris pour classer la 1ère fois: 0.477 s (IndexMemDicStorageTests.java:83)
2019-06-05T20:12 [INFO][TIMER] Temps pris pour classer la 2ème fois: 0.054 s (IndexMemDicStorageTests.java:86)
2019-06-05T20:12 [INFO] Recherche A OK ! Nb résultats selon A : 263 (IndexMemDicStorageTests.java:108)
2019-06-05T20:12 [INFO] Recherche B OK ! Nb résultats selon B : 263 (IndexMemDicStorageTests.java:111)
2019-06-05T20:12 [INFO] Recherche C OK ! Nb résultats selon C : 263 (IndexMemDicStorageTests.java:114)
2019-06-05T20:12 [INFO] resultsPositionsArray.length = 263 (IndexMemDicStorageTests.java:117)
2019-06-05T20:12 [INFO] Parsing de csvName = testdata/SMALL_100_000_yellow_tripdata_2015-04.csv (NewLoaderAndIndexTest.java:170)
2019-06-05T20:12 [INFO] PARSE : memusage init =  (ParallelLoader.java:112)
2019-06-05T20:12 [INFO][MEMORY] 190,485 (ParallelLoader.java:114)
2019-06-05T20:12 [INFO] Loader : nombre de résultats (total) lus = 200000   temps écoulé = 0.064 sactiveThreadNb = 0 (ParallelLoader.java:150)
2019-06-05T20:12 [ERROR] debugCount = 0 (ParallelLoader.java:203)
2019-06-05T20:12 [WARNING] Fail to save entry number 0: incorrect data (ParallelLoaderRunnable.java:287)
2019-06-05T20:12 [INFO] Parsing terminé !! temps écoulé = 0.908 s (ParallelLoader.java:217)
2019-06-05T20:12 [INFO] PARSE : Nombre de lignes (local) = 100000 (ParallelLoader.java:218)
2019-06-05T20:12 [INFO][TIMER] TEMPS TOTAL PRIS PAR TOUS LES PARSINGS: 0.917 s (NewLoaderAndIndexTest.java:189)
2019-06-05T20:12 [INFO] Bon nombre total de lignes ! nombre = 99999 (NewLoaderAndIndexTest.java:193)
2019-06-05T20:12 [INFO] Recherche A OK ! Nb résultats selon A : 263 (NewLoaderAndIndexTest.java:71)
2019-06-05T20:12 [INFO] Recherche B OK ! Nb résultats selon B : 263 (NewLoaderAndIndexTest.java:74)
2019-06-05T20:12 [INFO] Recherche C OK ! Nb résultats selon C : 263 (NewLoaderAndIndexTest.java:77)
2019-06-05T20:12 [INFO] resultsPositionsArray.length = 263 (NewLoaderAndIndexTest.java:80)
2019-06-05T20:12 [INFO] Recherche FLAG OK ! Nb résultats avec FLAG : 175  index len = 99999 (NewLoaderAndIndexTest.java:101)
2019-06-05T20:12 [INFO] Recherche STRUCT OK ! Nb résultats avec restructuration : 175  index len = 66666 (NewLoaderAndIndexTest.java:109)
2019-06-05T20:12 [INFO] setUpBeforeAll (IndexTreeMessyTest.java:73)
2019-06-05T20:12 [INFO] Nombre de résultats/entrées parsés INIT : 0 (IndexTreeMessyTest.java:186)
2019-06-05T20:12 [WARNING] Fail to save entry number 0: incorrect data (Loader.java:107)
2019-06-05T20:12 [WARNING] Fail to save entry number 0: incorrect data (Loader.java:107)
2019-06-05T20:12 [INFO] Loader : nombre de résultats (local) parsés = 0   temps écoulé = 0.57 millis (Loader.java:116)
2019-06-05T20:12 [INFO] Loader : nombre de résultats (local) parsés = 0   temps écoulé = 0.56 millis (Loader.java:116)
2019-06-05T20:12 [INFO][MEMORY] 401,153 (Loader.java:117)
2019-06-05T20:12 [INFO][MEMORY] 401,153 (Loader.java:117)
2019-06-05T20:12 [WARNING] Fail to save entry number 0: incorrect data (Loader.java:107)
2019-06-05T20:12 [INFO] Loader : nombre de résultats (local) parsés = 0   temps écoulé = 0.33 millis (Loader.java:116)
2019-06-05T20:12 [INFO][MEMORY] 273,252 (Loader.java:117)
2019-06-05T20:12 [WARNING] Fail to save entry number 0: incorrect data (Loader.java:107)
2019-06-05T20:12 [INFO] Loader : nombre de résultats (local) parsés = 0   temps écoulé = 1.97 millis (Loader.java:116)
2019-06-05T20:12 [INFO][MEMORY] 273,252 (Loader.java:117)
2019-06-05T20:13 [INFO] PARSE : FINAL USAGE (Loader.java:136)
2019-06-05T20:13 [INFO][MEMORY] 216,262 (Loader.java:138)
2019-06-05T20:13 [INFO] PARSE : FINAL USAGE (Loader.java:136)
2019-06-05T20:13 [INFO][MEMORY] 221,186 (Loader.java:138)
2019-06-05T20:13 [INFO] PARSE : FINAL USAGE (Loader.java:136)
2019-06-05T20:13 [INFO][MEMORY] 218,771 (Loader.java:138)
2019-06-05T20:13 [INFO] PARSE : FINAL USAGE (Loader.java:136)
2019-06-05T20:13 [INFO][MEMORY] 216,694 (Loader.java:138)
2019-06-05T20:13 [INFO][TIMER] TEMPS TOTAL pris par le parsing: 6.57 s (IndexTreeMessyTest.java:314)
2019-06-05T20:13 [INFO] setUpBeforeAll OK (IndexTreeMessyTest.java:322)
2019-06-05T20:13 [DEBUG] TableHandler.flushAllIndexOnDisk : size = 15 (Table.java:334)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,432 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@2bef09c0 (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@62ce72ff (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@58a63629 (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@7de843ef (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@2e1af49b (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@3fd2322d (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@56ac5c80 (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@69e49a81 (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@1023150a (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@3ece1e79 (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@ab20d7c (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@474fae39 (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@4f4c789f (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@13f9ad9 (Table.java:338)
2019-06-05T20:13 [INFO][MEMORY] IndexTreeDic.flushOnDisk 222,693 (IndexTreeDic.java:517)
2019-06-05T20:13 [INFO] Table.flushAllIndexOnDisk : flush de l'arbre !index.indexTree.IndexTreeDic@1b7cae6f (Table.java:338)
2019-06-05T20:13 [ERROR] SerialStructure.writeStructure :  impossible de sauvegarder la structure du disque. (SerialStructure.java:56)
2019-06-05T20:13 [ERROR] java.nio.HeapByteBuffer (SerialStructure.java:57)
2019-06-05T20:13 [ERROR] SerialStructure.loadStructure :  impossible de charger la structure du disque. (SerialStructure.java:73)
2019-06-05T20:13 [ERROR] null (SerialStructure.java:74)
2019-06-05T20:13 [INFO] Lancé (IndexTreeMessyTest.java:366)
2019-06-05T20:13 [INFO][TIMER] tempsPrisPourRecherchesSurFichiers: 48 millis (IndexTreeDic.java:1164)
2019-06-05T20:13 [INFO][TIMER] Temps total recherche: 0.051 s (IndexTreeMessyTest.java:408)
2019-06-05T20:13 [INFO][TIMER] Temps parcours des résultats: 1.72 millis (IndexTreeMessyTest.ja
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-15 06:10:39 +0000