MegaRAID SAS 2108 - SW RAID 10 (1+0 - 2 + 2 + 2)

START TESTS IO = cached WC = wt PC = on

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-on                  4000  4096    1  462.85 13.51%     0.025      122.28   0.00000  0.00000  3424
cached-wt-on                  4000  4096    2  609.24 18.90%     0.038      182.24   0.00000  0.00000  3222
cached-wt-on                  4000  4096    4  582.31 77.16%     0.077      440.85   0.00000  0.00000   755
cached-wt-on                  4000  4096    8  601.47 176.7%     0.149      880.28   0.00000  0.00000   340

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-on                  4000  4096    1    0.80 0.539%    14.649      127.61   0.00000  0.00000   148
cached-wt-on                  4000  4096    2    1.56 4.215%    14.876       98.72   0.00000  0.00000    37
cached-wt-on                  4000  4096    4    2.75 11.68%    15.010      114.78   0.00000  0.00000    24
cached-wt-on                  4000  4096    8    4.78 38.97%    17.886      172.75   0.00000  0.00000    12

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-on                  4000  4096    1  448.06 55.54%     0.026      286.06   0.00000  0.00000   807
cached-wt-on                  4000  4096    2  294.28 95.77%     0.078      639.71   0.00000  0.00000   307
cached-wt-on                  4000  4096    4  229.24 166.0%     0.199     1005.29   0.00000  0.00000   138
cached-wt-on                  4000  4096    8  209.42 310.5%     0.436     1184.45   0.00000  0.00000    67

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-on                  4000  4096    1    5.61     0%     0.009        0.06   0.00000  0.00000     0
cached-wt-on                  4000  4096    2    5.37     0%     0.009        0.06   0.00000  0.00000     0
cached-wt-on                  4000  4096    4    5.13 0.831%     0.010        0.07   0.00000  0.00000   617
cached-wt-on                  4000  4096    8    5.20 17.73%     0.017        0.10   0.00000  0.00000    29

START TESTS IO = cached WC = wt PC =

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-off                 4000  4096    1  475.14 16.39%     0.025       53.90   0.00000  0.00000  2898
cached-wt-off                 4000  4096    2  592.56 30.55%     0.039      217.23   0.00000  0.00000  1939
cached-wt-off                 4000  4096    4  574.67 84.23%     0.078      404.92   0.00000  0.00000   682
cached-wt-off                 4000  4096    8  599.66 179.3%     0.150      927.41   0.00000  0.00000   334

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-off                 4000  4096    1    0.80 0.964%    14.614      108.20   0.00000  0.00000    83
cached-wt-off                 4000  4096    2    1.53 4.133%    14.851      120.26   0.00000  0.00000    37
cached-wt-off                 4000  4096    4    2.83 12.06%    14.839       94.00   0.00000  0.00000    23
cached-wt-off                 4000  4096    8    4.78 37.66%    17.597      133.40   0.00000  0.00000    13

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-off                 4000  4096    1  405.33 54.45%     0.028      255.12   0.00000  0.00000   744
cached-wt-off                 4000  4096    2  243.62 83.19%     0.094      964.72   0.00000  0.00000   293
cached-wt-off                 4000  4096    4  182.77 118.7%     0.250     1421.38   0.00000  0.00000   154
cached-wt-off                 4000  4096    8  156.26 212.6%     0.585     1716.30   0.00000  0.00000    73

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-off                 4000  4096    1    5.84     0%     0.010        0.06   0.00000  0.00000     0
cached-wt-off                 4000  4096    2    5.61     0%     0.009        0.06   0.00000  0.00000     0
cached-wt-off                 4000  4096    4    5.68 0.774%     0.010        0.07   0.00000  0.00000   732
cached-wt-off                 4000  4096    8    5.74 24.38%     0.015        0.08   0.00000  0.00000    24

START TESTS IO = cached WC = awb PC = on

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-on                 4000  4096    1  418.05 15.62%     0.028      648.42   0.00000  0.00000  2676
cached-awb-on                 4000  4096    2  467.11 15.63%     0.049      798.03   0.00000  0.00000  2988
cached-awb-on                 4000  4096    4  475.21 47.65%     0.092     1026.38   0.00000  0.00000   997
cached-awb-on                 4000  4096    8  465.94 65.01%     0.184     1639.50   0.00000  0.00000   717

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-on                 4000  4096    1    0.84 0.810%    13.934      115.33   0.00000  0.00000   104
cached-awb-on                 4000  4096    2    1.58 4.261%    14.398      116.67   0.00000  0.00000    37
cached-awb-on                 4000  4096    4    2.93 12.62%    14.281      110.91   0.00000  0.00000    23
cached-awb-on                 4000  4096    8    4.68 31.20%    17.261      253.25   0.00000  0.00000    15

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-on                 4000  4096    1  498.96 61.54%     0.023      545.80   0.00000  0.00000   811
cached-awb-on                 4000  4096    2  497.93 155.0%     0.046      600.66   0.00000  0.00000   321
cached-awb-on                 4000  4096    4  484.76 339.7%     0.094      625.07   0.00000  0.00000   143
cached-awb-on                 4000  4096    8  482.63 971.8%     0.185      512.20   0.00000  0.00000    50

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-on                 4000  4096    1   11.13 0.189%     0.009        0.05   0.00000  0.00000  5859
cached-awb-on                 4000  4096    2   10.71     0%     0.009        0.06   0.00000  0.00000     0
cached-awb-on                 4000  4096    4   10.66 1.092%     0.010        0.08   0.00000  0.00000   977
cached-awb-on                 4000  4096    8   10.21 11.76%     0.017        0.09   0.00000  0.00000    87

START TESTS IO = cached WC = awb PC = off

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-off                4000  4096    1  412.57 14.48%     0.028      784.63   0.00000  0.00000  2849
cached-awb-off                4000  4096    2  518.59 19.50%     0.045      695.77   0.00000  0.00000  2659
cached-awb-off                4000  4096    4  520.66 47.19%     0.083     1043.65   0.00000  0.00000  1103
cached-awb-off                4000  4096    8  519.11 85.76%     0.166     1363.41   0.00000  0.00000   605

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-off                4000  4096    1    0.82 0.674%    14.227      158.82   0.00000  0.00000   122
cached-awb-off                4000  4096    2    1.55 4.241%    14.736      157.93   0.00000  0.00000    37
cached-awb-off                4000  4096    4    2.95 12.04%    14.513      120.55   0.00000  0.00000    25
cached-awb-off                4000  4096    8    4.94 31.88%    17.059      137.71   0.00000  0.00000    16

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-off                4000  4096    1  488.89 61.01%     0.023      456.56   0.00000  0.00000   801
cached-awb-off                4000  4096    2  491.26 149.0%     0.046      512.93   0.00000  0.00000   330
cached-awb-off                4000  4096    4  485.30 328.9%     0.094      529.83   0.00000  0.00000   148
cached-awb-off                4000  4096    8  455.13 925.3%     0.197      562.44   0.00000  0.00000    49

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-off                4000  4096    1    9.74 0.083%     0.009        0.05   0.00000  0.00000 11719
cached-awb-off                4000  4096    2    8.70     0%     0.009        0.07   0.00000  0.00000     0
cached-awb-off                4000  4096    4    9.30 3.175%     0.010        0.09   0.00000  0.00000   293
cached-awb-off                4000  4096    8    8.25 9.569%     0.016        0.10   0.00000  0.00000    86

START TESTS IO = direct WC = wt PC = on

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-on                  4000  4096    1  470.31 13.90%     0.025      122.05   0.00000  0.00000  3382
direct-wt-on                  4000  4096    2  608.38 20.54%     0.038      230.56   0.00000  0.00000  2961
direct-wt-on                  4000  4096    4  593.23 79.81%     0.073      322.20   0.00000  0.00000   743
direct-wt-on                  4000  4096    8  602.60 171.7%     0.146      984.91   0.00000  0.00000   351

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-on                  4000  4096    1    0.79 0.695%    14.800      127.98   0.00000  0.00000   114
direct-wt-on                  4000  4096    2    1.41 3.943%    16.363      126.50   0.00000  0.00000    36
direct-wt-on                  4000  4096    4    2.72 11.50%    15.148      215.17   0.00000  0.00000    24
direct-wt-on                  4000  4096    8    4.62 38.64%    18.004      153.71   0.00000  0.00000    12

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-on                  4000  4096    1  458.64 57.13%     0.025      230.74   0.00000  0.00000   803
direct-wt-on                  4000  4096    2  290.16 94.70%     0.079      824.62   0.00000  0.00000   306
direct-wt-on                  4000  4096    4  229.99 148.4%     0.199      923.75   0.00000  0.00000   155
direct-wt-on                  4000  4096    8  208.63 313.7%     0.433     1068.90   0.00000  0.00000    66

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-on                  4000  4096    1    5.28 0.090%     0.009        0.06   0.00000  0.00000  5859
direct-wt-on                  4000  4096    2    6.08 0.103%     0.009        0.05   0.00000  0.00000  5859
direct-wt-on                  4000  4096    4    6.40 3.657%     0.010        0.08   0.00000  0.00000   175
direct-wt-on                  4000  4096    8    6.17 8.685%     0.017        0.15   0.00000  0.00000    71

START TESTS IO = direct WC = wt PC = off

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-off                 4000  4096    1  467.03 17.57%     0.025      132.30   0.00000  0.00000  2657
direct-wt-off                 4000  4096    2  611.99 34.98%     0.038      211.65   0.00000  0.00000  1749
direct-wt-off                 4000  4096    4  589.11 85.15%     0.075      356.54   0.00000  0.00000   692
direct-wt-off                 4000  4096    8  597.58 180.5%     0.147      892.34   0.00000  0.00000   331

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-off                 4000  4096    1    0.79 0.862%    14.831      100.28   0.00000  0.00000    92
direct-wt-off                 4000  4096    2    1.51 4.213%    15.191      115.55   0.00000  0.00000    36
direct-wt-off                 4000  4096    4    2.72 11.66%    15.295      106.87   0.00000  0.00000    23
direct-wt-off                 4000  4096    8    4.55 37.06%    18.315      142.48   0.00000  0.00000    12

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-off                 4000  4096    1  397.09 52.86%     0.029      239.47   0.00000  0.00000   751
direct-wt-off                 4000  4096    2  240.42 81.34%     0.095     1297.71   0.00000  0.00000   296
direct-wt-off                 4000  4096    4  181.09 126.6%     0.249     1506.76   0.00000  0.00000   143
direct-wt-off                 4000  4096    8  157.02 205.9%     0.584     1870.55   0.00000  0.00000    76

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-off                 4000  4096    1    5.95 0.050%     0.010        0.06   0.00000  0.00000 11719
direct-wt-off                 4000  4096    2    6.00     0%     0.009        0.05   0.00000  0.00000     0
direct-wt-off                 4000  4096    4    5.79 1.582%     0.010        0.06   0.00000  0.00000   366
direct-wt-off                 4000  4096    8    5.94 32.05%     0.017        0.07   0.00000  0.00000    19

START TESTS IO = direct WC = awb PC = on

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-on                 4000  4096    1  409.87 16.53%     0.028      628.68   0.00000  0.00000  2479
direct-awb-on                 4000  4096    2  479.43 17.85%     0.048      546.33   0.00000  0.00000  2686
direct-awb-on                 4000  4096    4  469.68 48.44%     0.094     1023.92   0.00000  0.00000   970
direct-awb-on                 4000  4096    8  470.86 52.61%     0.185     1559.41   0.00000  0.00000   895

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-on                 4000  4096    1    0.81 0.773%    14.482      143.94   0.00000  0.00000   105
direct-awb-on                 4000  4096    2    1.60 4.115%    14.414       96.66   0.00000  0.00000    39
direct-awb-on                 4000  4096    4    2.95 13.23%    14.302       90.20   0.00000  0.00000    22
direct-awb-on                 4000  4096    8    4.91 26.20%    17.121      146.82   0.00000  0.00000    19

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-on                 4000  4096    1  496.21 61.84%     0.023       51.54   0.00000  0.00000   802
direct-awb-on                 4000  4096    2  500.74 151.4%     0.046      346.99   0.00000  0.00000   331
direct-awb-on                 4000  4096    4  493.30 326.0%     0.092      367.83   0.00000  0.00000   151
direct-awb-on                 4000  4096    8  482.48 972.3%     0.185      394.22   0.00000  0.00000    50

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-on                 4000  4096    1   12.62 0.107%     0.009        0.07   0.00000  0.00000 11719
direct-awb-on                 4000  4096    2   12.54 0.213%     0.009        0.06   0.00000  0.00000  5859
direct-awb-on                 4000  4096    4   11.28 3.466%     0.010        0.07   0.00000  0.00000   325
direct-awb-on                 4000  4096    8    9.51 4.543%     0.017        0.09   0.00000  0.00000   209

START TESTS IO = direct WC = awb PC = off

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-off                4000  4096    1  419.91 14.83%     0.028      836.43   0.00000  0.00000  2830
direct-awb-off                4000  4096    2  508.28 20.26%     0.045      741.52   0.00000  0.00000  2508
direct-awb-off                4000  4096    4  515.65 57.17%     0.087     1018.63   0.00000  0.00000   902
direct-awb-off                4000  4096    8  519.17 90.11%     0.168     1168.69   0.00000  0.00000   576

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-off                4000  4096    1    0.83 0.620%    14.187      122.07   0.00000  0.00000   133
direct-awb-off                4000  4096    2    1.57 3.062%    14.386      145.25   0.00000  0.00000    51
direct-awb-off                4000  4096    4    2.93 12.51%    14.586      132.94   0.00000  0.00000    23
direct-awb-off                4000  4096    8    4.81 31.41%    17.062      138.90   0.00000  0.00000    15

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-off                4000  4096    1  489.41 64.14%     0.023       98.94   0.00000  0.00000   763
direct-awb-off                4000  4096    2  475.58 145.4%     0.048      173.13   0.00000  0.00000   327
direct-awb-off                4000  4096    4  463.76 301.8%     0.098      239.71   0.00000  0.00000   154
direct-awb-off                4000  4096    8  440.92 875.4%     0.205      537.00   0.00000  0.00000    50

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-off                4000  4096    1    9.07     0%     0.010        0.07   0.00000  0.00000     0
direct-awb-off                4000  4096    2    9.85 0.168%     0.009        0.06   0.00000  0.00000  5859
direct-awb-off                4000  4096    4    9.51 3.894%     0.010        0.09   0.00000  0.00000   244
direct-awb-off                4000  4096    8    8.65 6.868%     0.018        0.11   0.00000  0.00000   126

Skripts

#!/bin/bash
storcli=/opt/MegaRAID/storcli/storcli64

$storcli /c0/vall set iopolicy=cached
$storcli /c0/vall set wrcache=awb
$storcli /c0/vall set pdcache=on

LOGFILE=/root/tio.md.txt

for iopolicy  in cached  direct
do
 $storcli /c0/vall set iopolicy=$iopolicy
        IO=$iopolicy 
        echo = IO=$IO

 for wrcache in wt awb
 do
  $storcli /c0/vall set wrcache=$wrcache
  WC=$wrcache
  echo == WC=$WC
  for pdcache in on off
  do
   $storcli /c0/vall set pdcache=$pdcache
   PC=$pdcache
   echo === PC=$PC
   echo 
   IDENT=$IO-$WC-$PC
   $storcli /c0/vall show | grep RAID | tee -a $LOGFILE
   echo  ==== START TESTS IO = $IO    WC = $WC    PC = $PC    =====  `date +%T` ==== | tee -a $LOGFILE
   sleep 3 
   tiobench --size 4000 --ident $IDENT --dir=/mnt/test/ --numruns 3 | tee -a $LOGFILE
  done
 done
done

MegaRAID SAS 2108 - HW RAID 10 (2 + 2 + 2)

Base Board Information
        Manufacturer: Supermicro
        Product Name: X9DRi-LN4+/X9DR3-LN4+

q

RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05)
SAS 9260CV-8i
Sistēma startēta ar parametru "mem = 1G"
FS - EXT4

HW RAID 10 

2 + 2 + 2

RWTD

storcli /c0/v0 show 

----------------------------------------------------------
DG/VD TYPE   State Access Consist Cache sCC     Size Name 
----------------------------------------------------------
0/0   RAID10 Optl  RW     No      RWTD  -   5.454 TB      
----------------------------------------------------------

root@ubuntu:~# tiobench --size 2000 --identifier R10-RWTD --numruns 3 
Run #3: /usr/bin/tiotest -t 8 -f 250 -r 500 -b 4096 -d . -TTT
Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD                      2000  4096    1  164.01 10.61%     0.071      110.05   0.00000  0.00000  1545
R10-RWTD                      2000  4096    2  136.11 16.77%     0.168      149.97   0.00000  0.00000   811
R10-RWTD                      2000  4096    4  127.26 36.09%     0.346      310.12   0.00000  0.00000   353
R10-RWTD                      2000  4096    8  143.80 82.40%     0.600      512.41   0.00000  0.00000   174

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD                      2000  4096    1    1.05 0.510%    11.167      103.91   0.00000  0.00000   206
R10-RWTD                      2000  4096    2    1.86 3.086%    10.919      100.91   0.00000  0.00000    60
R10-RWTD                      2000  4096    4    3.40 6.955%    11.365      119.16   0.00000  0.00000    49
R10-RWTD                      2000  4096    8    5.30 20.34%    13.479      241.22   0.00000  0.00000    26

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD                      2000  4096    1  244.78 33.16%     0.046      963.48   0.00000  0.00000   738
R10-RWTD                      2000  4096    2  199.02 62.89%     0.112      758.66   0.00000  0.00000   316
R10-RWTD                      2000  4096    4  186.14 117.4%     0.241      560.05   0.00000  0.00000   159
R10-RWTD                      2000  4096    8  177.05 216.8%     0.508      711.64   0.00000  0.00000    82

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD                      2000  4096    1    8.06     0%     0.009        0.06   0.00000  0.00000     0
R10-RWTD                      2000  4096    2    7.78     0%     0.009        0.04   0.00000  0.00000     0
R10-RWTD                      2000  4096    4    7.82     0%     0.010        0.08   0.00000  0.00000     0
R10-RWTD                      2000  4096    8    7.90 9.166%     0.017        0.09   0.00000  0.00000    86

RWTC

storcli /c0/v0 set iopolicy=cached
storcli /c0/v0 show

----------------------------------------------------------
DG/VD TYPE   State Access Consist Cache sCC     Size Name 
----------------------------------------------------------
0/0   RAID10 Optl  RW     No      RWTC  -   5.454 TB      
----------------------------------------------------------

root@ubuntu:~# tiobench --size 2000 --identifier R10-RWTC --numruns 3 
Run #3: /usr/bin/tiotest -t 8 -f 250 -r 500 -b 4096 -d . -TTT

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC                      2000  4096    1  169.42 10.76%     0.069      108.65   0.00000  0.00000  1574
R10-RWTC                      2000  4096    2  139.71 17.76%     0.165      162.62   0.00000  0.00000   787
R10-RWTC                      2000  4096    4  133.68 37.36%     0.325      237.25   0.00000  0.00000   358
R10-RWTC                      2000  4096    8  132.39 75.94%     0.645      637.03   0.00000  0.00000   174

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC                      2000  4096    1    1.04 0.694%    11.234       84.40   0.00000  0.00000   150
R10-RWTC                      2000  4096    2    1.98 3.624%    11.001       82.62   0.00000  0.00000    55
R10-RWTC                      2000  4096    4    3.36 5.441%    11.392      165.62   0.00000  0.00000    62
R10-RWTC                      2000  4096    8    5.25 19.31%    13.533      180.47   0.00000  0.00000    27

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC                      2000  4096    1  247.02 30.99%     0.045      322.56   0.00000  0.00000   797
R10-RWTC                      2000  4096    2  203.29 67.52%     0.110      563.23   0.00000  0.00000   301
R10-RWTC                      2000  4096    4  185.36 116.4%     0.241      526.25   0.00000  0.00000   159
R10-RWTC                      2000  4096    8  177.23 225.7%     0.506      616.67   0.00000  0.00000    78

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC                      2000  4096    1    7.41     0%     0.009        0.06   0.00000  0.00000     0
R10-RWTC                      2000  4096    2    7.98     0%     0.009        0.05   0.00000  0.00000     0
R10-RWTC                      2000  4096    4    7.76 1.059%     0.010        0.08   0.00000  0.00000   732
R10-RWTC                      2000  4096    8    7.92 8.446%     0.017        0.07   0.00000  0.00000    94

RWTC - Disk cache enabled

tiobench --size 2000 --identifier R10-RWTC 
Run #1: /usr/bin/tiotest -t 8 -f 250 -r 500 -b 4096 -d . -TTT

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC                      2000  4096    1  222.69 15.58%     0.017       24.78   0.00000  0.00000  1428
R10-RWTC                      2000  4096    2  126.77 15.46%     0.061       58.48   0.00000  0.00000   820
R10-RWTC                      2000  4096    4  162.10 44.19%     0.087       66.98   0.00000  0.00000   367
R10-RWTC                      2000  4096    8  126.15 72.59%     0.230      208.33   0.00000  0.00000   174

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC                      2000  4096    1    1.09 0.694%     3.596       27.37   0.00000  0.00000   156
R10-RWTC                      2000  4096    2    2.01 1.441%     3.693       25.86   0.00000  0.00000   139
R10-RWTC                      2000  4096    4    2.94 3.841%     3.876       28.29   0.00000  0.00000    77
R10-RWTC                      2000  4096    8    5.25 17.22%     4.559       70.20   0.00000  0.00000    31

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC                      2000  4096    1  281.20 36.22%     0.013      174.99   0.00000  0.00000   776
R10-RWTC                      2000  4096    2  199.49 64.79%     0.037      171.67   0.00000  0.00000   308
R10-RWTC                      2000  4096    4  188.08 119.7%     0.079      175.89   0.00000  0.00000   157
R10-RWTC                      2000  4096    8  176.24 208.0%     0.169      256.70   0.00000  0.00000    85

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC                      2000  4096    1    8.10     0%     0.003        0.03   0.00000  0.00000     0
R10-RWTC                      2000  4096    2    7.89     0%     0.003        0.02   0.00000  0.00000     0
R10-RWTC                      2000  4096    4    7.79     0%     0.003        0.03   0.00000  0.00000     0
R10-RWTC                      2000  4096    8    7.93 6.496%     0.006        0.03   0.00000  0.00000   122

RWTC - Disk cache disabled

# tiobench --size 2000 --identifier R10-RWTC-PdCacOff 
Run #1: /usr/bin/tiotest -t 8 -f 250 -r 500 -b 4096 -d . -TTT

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC-PdCacOff             2000  4096    1  221.14 14.06%     0.018       23.75   0.00000  0.00000  1572
R10-RWTC-PdCacOff             2000  4096    2  122.06 15.99%     0.063       50.35   0.00000  0.00000   763
R10-RWTC-PdCacOff             2000  4096    4  131.95 35.99%     0.108       83.59   0.00000  0.00000   367
R10-RWTC-PdCacOff             2000  4096    8  130.39 73.15%     0.219      183.32   0.00000  0.00000   178

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC-PdCacOff             2000  4096    1    1.11 0.509%     3.532       39.69   0.00000  0.00000   217
R10-RWTC-PdCacOff             2000  4096    2    2.11 3.894%     3.485       26.37   0.00000  0.00000    54
R10-RWTC-PdCacOff             2000  4096    4    2.99 3.209%     3.787       32.58   0.00000  0.00000    93
R10-RWTC-PdCacOff             2000  4096    8    4.89 17.40%     4.410       82.89   0.00000  0.00000    28

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC-PdCacOff             2000  4096    1  218.60 29.25%     0.017      237.86   0.00000  0.00000   747
R10-RWTC-PdCacOff             2000  4096    2  171.45 56.99%     0.043      209.19   0.00000  0.00000   301
R10-RWTC-PdCacOff             2000  4096    4  157.12 102.8%     0.095      250.64   0.00000  0.00000   153
R10-RWTC-PdCacOff             2000  4096    8  138.59 166.9%     0.209      340.25   0.00000  0.00000    83

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTC-PdCacOff             2000  4096    1    6.02     0%     0.003        0.02   0.00000  0.00000     0
R10-RWTC-PdCacOff             2000  4096    2    5.87     0%     0.003        0.02   0.00000  0.00000     0
R10-RWTC-PdCacOff             2000  4096    4    5.98     0%     0.003        0.03   0.00000  0.00000     0
R10-RWTC-PdCacOff             2000  4096    8    5.99 6.137%     0.006        0.05   0.00000  0.00000    98

RWTD - Disk cache disabled

root@ubuntu:~# tiobench --size 2000 --identifier R10-RWTD-PdCacOff 
Run #1: /usr/bin/tiotest -t 8 -f 250 -r 500 -b 4096 -d . -TTT

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD-PdCacOff             2000  4096    1  225.59 13.71%     0.017       23.27   0.00000  0.00000  1645
R10-RWTD-PdCacOff             2000  4096    2  128.78 18.03%     0.060       74.44   0.00000  0.00000   714
R10-RWTD-PdCacOff             2000  4096    4  120.24 34.48%     0.125      108.35   0.00000  0.00000   349
R10-RWTD-PdCacOff             2000  4096    8  150.00 78.87%     0.178      183.56   0.00000  0.00000   190

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD-PdCacOff             2000  4096    1    1.07 0.518%     3.662       28.25   0.00000  0.00000   206
R10-RWTD-PdCacOff             2000  4096    2    2.12 3.794%     3.478       32.29   0.00000  0.00000    56
R10-RWTD-PdCacOff             2000  4096    4    3.92 5.926%     3.850       33.38   0.00000  0.00000    66
R10-RWTD-PdCacOff             2000  4096    8    5.40 14.65%     4.195       54.68   0.00000  0.00000    37

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD-PdCacOff             2000  4096    1  211.89 27.29%     0.018      231.49   0.00000  0.00000   776
R10-RWTD-PdCacOff             2000  4096    2  176.41 58.00%     0.042      224.31   0.00000  0.00000   304
R10-RWTD-PdCacOff             2000  4096    4  156.11 97.60%     0.094      261.58   0.00000  0.00000   160
R10-RWTD-PdCacOff             2000  4096    8  137.09 164.4%     0.209      325.76   0.00000  0.00000    83

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD-PdCacOff             2000  4096    1    6.12     0%     0.003        0.02   0.00000  0.00000     0
R10-RWTD-PdCacOff             2000  4096    2    6.24     0%     0.003        0.02   0.00000  0.00000     0
R10-RWTD-PdCacOff             2000  4096    4    6.00     0%     0.003        0.02   0.00000  0.00000     0
R10-RWTD-PdCacOff             2000  4096    8    6.05 4.957%     0.006        0.04   0.00000  0.00000   122

RWTD - Disk cache enabled

tiobench --size 2000 --identifier R10-RWTD-PdCacOn 
Run #1: /usr/bin/tiotest -t 8 -f 250 -r 500 -b 4096 -d . -TTT

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD-PdCacOn              2000  4096    1  222.84 14.61%     0.017       20.89   0.00000  0.00000  1524
R10-RWTD-PdCacOn              2000  4096    2  129.22 15.81%     0.060       48.82   0.00000  0.00000   817
R10-RWTD-PdCacOn              2000  4096    4  131.79 38.19%     0.110       83.58   0.00000  0.00000   345
R10-RWTD-PdCacOn              2000  4096    8  141.00 75.24%     0.188      151.67   0.00000  0.00000   187

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD-PdCacOn              2000  4096    1    1.06 0.703%     3.697       27.83   0.00000  0.00000   150
R10-RWTD-PdCacOn              2000  4096    2    1.99 0.970%     3.814       25.99   0.00000  0.00000   206
R10-RWTD-PdCacOn              2000  4096    4    2.88 6.938%     4.034       51.90   0.00000  0.00000    42
R10-RWTD-PdCacOn              2000  4096    8    5.36 14.82%     3.940       54.28   0.00000  0.00000    36

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD-PdCacOn              2000  4096    1  280.53 34.56%     0.013      205.93   0.00000  0.00000   812
R10-RWTD-PdCacOn              2000  4096    2  199.38 62.92%     0.037      165.82   0.00000  0.00000   317
R10-RWTD-PdCacOn              2000  4096    4  191.24 96.84%     0.079      157.06   0.00000  0.00000   197
R10-RWTD-PdCacOn              2000  4096    8  175.74 206.3%     0.170      391.90   0.00000  0.00000    85

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RWTD-PdCacOn              2000  4096    1    7.98 0.204%     0.003        0.01   0.00000  0.00000  3906
R10-RWTD-PdCacOn              2000  4096    2    7.78     0%     0.003        0.01   0.00000  0.00000     0
R10-RWTD-PdCacOn              2000  4096    4    8.24 1.687%     0.003        0.03   0.00000  0.00000   488
R10-RWTD-PdCacOn              2000  4096    8    8.10 11.61%     0.006        0.03   0.00000  0.00000    70

Always Write Back

   storcli /c0/v0 set wrcache=AWB 

RAWBD PdCacOn

storcli /c0/v0 show
----------------------------------------------------------
DG/VD TYPE   State Access Consist Cache sCC     Size Name 
----------------------------------------------------------
0/0   RAID10 Optl  RW     No      RAWBD -   5.454 TB      
----------------------------------------------------------

tiobench --size 2000 --identifier R10-RAWBD-PdCacOn  --numruns 3
Run #3: /usr/bin/tiotest -t 8 -f 250 -r 500 -b 4096 -d . -TTT

Unit information
================
File size = megabytes
Blk Size  = bytes
Rate      = megabytes per second
CPU%      = percentage of CPU used during the test
Latency   = milliseconds
Lat%      = percent of requests that took longer than X seconds
CPU Eff   = Rate divided by CPU% - throughput per cpu load

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBD-PdCacOn             2000  4096    1  146.53 9.671%     0.080     1344.87   0.00000  0.00000  1515
R10-RAWBD-PdCacOn             2000  4096    2  143.45 16.55%     0.158     1364.29   0.00000  0.00000   866
R10-RAWBD-PdCacOn             2000  4096    4  119.89 32.65%     0.362     1940.91   0.00000  0.00000   367
R10-RAWBD-PdCacOn             2000  4096    8  127.14 70.38%     0.668     1781.70   0.00000  0.00000   181

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBD-PdCacOn             2000  4096    1    1.17 0.616%    10.053      117.77   0.00000  0.00000   189
R10-RAWBD-PdCacOn             2000  4096    2    1.94 3.273%    10.045       85.48   0.00000  0.00000    59
R10-RAWBD-PdCacOn             2000  4096    4    3.43 5.976%    10.275      119.88   0.00000  0.00000    57
R10-RAWBD-PdCacOn             2000  4096    8    6.03 18.00%    11.818      185.64   0.00000  0.00000    33

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBD-PdCacOn             2000  4096    1  293.70 42.82%     0.039      252.09   0.00000  0.00000   686
R10-RAWBD-PdCacOn             2000  4096    2  263.84 83.16%     0.086      353.42   0.00000  0.00000   317
R10-RAWBD-PdCacOn             2000  4096    4  275.19 165.3%     0.163      474.69   0.00000  0.00000   166
R10-RAWBD-PdCacOn             2000  4096    8  257.68 351.0%     0.343     1178.33   0.00000  0.00000    73

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBD-PdCacOn             2000  4096    1   14.03     0%     0.009        0.08   0.00000  0.00000     0
R10-RAWBD-PdCacOn             2000  4096    2   15.38 1.312%     0.008        0.04   0.00000  0.00000  1172
R10-RAWBD-PdCacOn             2000  4096    4   14.37     0%     0.010        0.07   0.00000  0.00000     0
R10-RAWBD-PdCacOn             2000  4096    8   18.77 9.612%     0.017        0.10   0.00000  0.00000   195

RAWBC - Disk Cache On

storcli /c0/v0 set iopolicy=cached
storcli /c0/v0 show

----------------------------------------------------------
DG/VD TYPE   State Access Consist Cache sCC     Size Name 
----------------------------------------------------------
0/0   RAID10 Optl  RW     No      RAWBC -   5.454 TB      
----------------------------------------------------------

tiobench --size 2000 --identifier R10-RAWBC-PdCacOn  --numruns 3
Run #3: /usr/bin/tiotest -t 8 -f 250 -r 500 -b 4096 -d . -TTT

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBC-PdCacOn             2000  4096    1  146.89 9.313%     0.080      986.74   0.00000  0.00000  1577
R10-RAWBC-PdCacOn             2000  4096    2  124.14 14.54%     0.185     1406.61   0.00000  0.00000   853
R10-RAWBC-PdCacOn             2000  4096    4  125.12 33.76%     0.347     1811.97   0.00000  0.00000   371
R10-RAWBC-PdCacOn             2000  4096    8  126.05 70.05%     0.679     1398.69   0.00000  0.00000   180

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBC-PdCacOn             2000  4096    1    1.19 0.617%     9.882       96.18   0.00000  0.00000   192
R10-RAWBC-PdCacOn             2000  4096    2    2.02 2.255%    10.182       95.31   0.00000  0.00000    89
R10-RAWBC-PdCacOn             2000  4096    4    3.33 4.808%    10.761      140.33   0.00000  0.00000    69
R10-RAWBC-PdCacOn             2000  4096    8    5.98 17.81%    12.368      173.46   0.00000  0.00000    34

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBC-PdCacOn             2000  4096    1  295.63 39.42%     0.038      280.16   0.00000  0.00000   750
R10-RAWBC-PdCacOn             2000  4096    2  258.20 79.84%     0.088      190.60   0.00000  0.00000   323
R10-RAWBC-PdCacOn             2000  4096    4  282.15 140.9%     0.160      848.78   0.00000  0.00000   200
R10-RAWBC-PdCacOn             2000  4096    8  255.79 340.0%     0.342      400.85   0.00000  0.00000    75

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBC-PdCacOn             2000  4096    1   16.55     0%     0.009        0.06   0.00000  0.00000     0
R10-RAWBC-PdCacOn             2000  4096    2   18.37     0%     0.008        0.06   0.00000  0.00000     0
R10-RAWBC-PdCacOn             2000  4096    4   13.72 1.873%     0.010        0.07   0.00000  0.00000   732
R10-RAWBC-PdCacOn             2000  4096    8   13.51 12.91%     0.016        0.09   0.00000  0.00000   105

RAWBC - Disk Cache Off

 storcli /c0/v0 set pdcache=Off  

tiobench --size 2000 --identifier R10-RAWBC-PdCacOff  --numruns 1
Run #1: /usr/bin/tiotest -t 8 -f 250 -r 500 -b 4096 -d . -TTT

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBC-PdCacOff            2000  4096    1  178.90 10.80%     0.022      198.66   0.00000  0.00000  1656
R10-RAWBC-PdCacOff            2000  4096    2  111.23 12.90%     0.069      331.54   0.00000  0.00000   862
R10-RAWBC-PdCacOff            2000  4096    4  173.12 48.85%     0.087      402.51   0.00000  0.00000   354
R10-RAWBC-PdCacOff            2000  4096    8  135.25 71.82%     0.202      400.28   0.00000  0.00000   188

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBC-PdCacOff            2000  4096    1    1.14 0.552%     3.440       29.15   0.00000  0.00000   206
R10-RAWBC-PdCacOff            2000  4096    2    2.11 3.455%     3.378       51.59   0.00000  0.00000    61
R10-RAWBC-PdCacOff            2000  4096    4    3.84 3.635%     3.434       31.46   0.00000  0.00000   106
R10-RAWBC-PdCacOff            2000  4096    8    5.95 20.72%     3.958       53.06   0.00000  0.00000    29

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBC-PdCacOff            2000  4096    1  277.00 34.90%     0.013      320.74   0.00000  0.00000   794
R10-RAWBC-PdCacOff            2000  4096    2  211.19 67.75%     0.036      133.28   0.00000  0.00000   312
R10-RAWBC-PdCacOff            2000  4096    4  279.79 164.3%     0.053      116.90   0.00000  0.00000   170
R10-RAWBC-PdCacOff            2000  4096    8  212.86 254.1%     0.140      279.91   0.00000  0.00000    84

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-RAWBC-PdCacOff            2000  4096    1   15.27     0%     0.003        0.02   0.00000  0.00000     0
R10-RAWBC-PdCacOff            2000  4096    2   14.53 0.744%     0.003        0.03   0.00000  0.00000  1953
R10-RAWBC-PdCacOff            2000  4096    4   18.08     0%     0.003        0.03   0.00000  0.00000     0
R10-RAWBC-PdCacOff            2000  4096    8   15.72 9.660%     0.006        0.03   0.00000  0.00000   163

NRAWBC - Disk Cache Off

storcli /c0/v0 set rdcache=nora
storcli /c0/v0 show
-----------------------------------------------------------
DG/VD TYPE   State Access Consist Cache  sCC     Size Name 
-----------------------------------------------------------
0/0   RAID10 Optl  RW     No      NRAWBC -   5.454 TB      
-----------------------------------------------------------

tiobench --size 2000 --identifier R10-NRAWBC-PdCacOff  --numruns 1
Run #1: /usr/bin/tiotest -t 8 -f 250 -r 500 -b 4096 -d . -TTT

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-NRAWBC-PdCacOff           2000  4096    1  190.21 11.41%     0.020      433.66   0.00000  0.00000  1667
R10-NRAWBC-PdCacOff           2000  4096    2  109.12 14.42%     0.071      275.32   0.00000  0.00000   756
R10-NRAWBC-PdCacOff           2000  4096    4  123.45 33.13%     0.110      373.31   0.00000  0.00000   373
R10-NRAWBC-PdCacOff           2000  4096    8  128.07 70.72%     0.219      350.40   0.00000  0.00000   181

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-NRAWBC-PdCacOff           2000  4096    1    1.22 0.591%     3.210       61.00   0.00000  0.00000   206
R10-NRAWBC-PdCacOff           2000  4096    2    2.17 3.770%     3.446       48.13   0.00000  0.00000    57
R10-NRAWBC-PdCacOff           2000  4096    4    3.28 4.874%     3.360       34.23   0.00000  0.00000    67
R10-NRAWBC-PdCacOff           2000  4096    8    6.04 17.45%     3.928       48.08   0.00000  0.00000    35

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-NRAWBC-PdCacOff           2000  4096    1  288.05 39.86%     0.013      267.42   0.00000  0.00000   722
R10-NRAWBC-PdCacOff           2000  4096    2  216.66 73.93%     0.035       59.29   0.00000  0.00000   293
R10-NRAWBC-PdCacOff           2000  4096    4  262.35 133.5%     0.057      122.33   0.00000  0.00000   196
R10-NRAWBC-PdCacOff           2000  4096    8  213.97 265.4%     0.138      175.91   0.00000  0.00000    81

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
R10-NRAWBC-PdCacOff           2000  4096    1   14.42     0%     0.003        0.02   0.00000  0.00000     0
R10-NRAWBC-PdCacOff           2000  4096    2   17.76     0%     0.003        0.02   0.00000  0.00000     0
R10-NRAWBC-PdCacOff           2000  4096    4   15.63     0%     0.003        0.03   0.00000  0.00000     0
R10-NRAWBC-PdCacOff           2000  4096    8   15.65 16.42%     0.006        0.03   0.00000  0.00000    95