MegaRAID SAS 2108 - SW RAID 10 (1+0 - 2 + 2 + 2)

START TESTS IO = cached WC = wt PC = on

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-on                  4000  4096    1  462.85 13.51%     0.025      122.28   0.00000  0.00000  3424
cached-wt-on                  4000  4096    2  609.24 18.90%     0.038      182.24   0.00000  0.00000  3222
cached-wt-on                  4000  4096    4  582.31 77.16%     0.077      440.85   0.00000  0.00000   755
cached-wt-on                  4000  4096    8  601.47 176.7%     0.149      880.28   0.00000  0.00000   340

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-on                  4000  4096    1    0.80 0.539%    14.649      127.61   0.00000  0.00000   148
cached-wt-on                  4000  4096    2    1.56 4.215%    14.876       98.72   0.00000  0.00000    37
cached-wt-on                  4000  4096    4    2.75 11.68%    15.010      114.78   0.00000  0.00000    24
cached-wt-on                  4000  4096    8    4.78 38.97%    17.886      172.75   0.00000  0.00000    12

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-on                  4000  4096    1  448.06 55.54%     0.026      286.06   0.00000  0.00000   807
cached-wt-on                  4000  4096    2  294.28 95.77%     0.078      639.71   0.00000  0.00000   307
cached-wt-on                  4000  4096    4  229.24 166.0%     0.199     1005.29   0.00000  0.00000   138
cached-wt-on                  4000  4096    8  209.42 310.5%     0.436     1184.45   0.00000  0.00000    67

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-on                  4000  4096    1    5.61     0%     0.009        0.06   0.00000  0.00000     0
cached-wt-on                  4000  4096    2    5.37     0%     0.009        0.06   0.00000  0.00000     0
cached-wt-on                  4000  4096    4    5.13 0.831%     0.010        0.07   0.00000  0.00000   617
cached-wt-on                  4000  4096    8    5.20 17.73%     0.017        0.10   0.00000  0.00000    29

START TESTS IO = cached WC = wt PC =

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-off                 4000  4096    1  475.14 16.39%     0.025       53.90   0.00000  0.00000  2898
cached-wt-off                 4000  4096    2  592.56 30.55%     0.039      217.23   0.00000  0.00000  1939
cached-wt-off                 4000  4096    4  574.67 84.23%     0.078      404.92   0.00000  0.00000   682
cached-wt-off                 4000  4096    8  599.66 179.3%     0.150      927.41   0.00000  0.00000   334

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-off                 4000  4096    1    0.80 0.964%    14.614      108.20   0.00000  0.00000    83
cached-wt-off                 4000  4096    2    1.53 4.133%    14.851      120.26   0.00000  0.00000    37
cached-wt-off                 4000  4096    4    2.83 12.06%    14.839       94.00   0.00000  0.00000    23
cached-wt-off                 4000  4096    8    4.78 37.66%    17.597      133.40   0.00000  0.00000    13

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-off                 4000  4096    1  405.33 54.45%     0.028      255.12   0.00000  0.00000   744
cached-wt-off                 4000  4096    2  243.62 83.19%     0.094      964.72   0.00000  0.00000   293
cached-wt-off                 4000  4096    4  182.77 118.7%     0.250     1421.38   0.00000  0.00000   154
cached-wt-off                 4000  4096    8  156.26 212.6%     0.585     1716.30   0.00000  0.00000    73

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-wt-off                 4000  4096    1    5.84     0%     0.010        0.06   0.00000  0.00000     0
cached-wt-off                 4000  4096    2    5.61     0%     0.009        0.06   0.00000  0.00000     0
cached-wt-off                 4000  4096    4    5.68 0.774%     0.010        0.07   0.00000  0.00000   732
cached-wt-off                 4000  4096    8    5.74 24.38%     0.015        0.08   0.00000  0.00000    24

START TESTS IO = cached WC = awb PC = on

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-on                 4000  4096    1  418.05 15.62%     0.028      648.42   0.00000  0.00000  2676
cached-awb-on                 4000  4096    2  467.11 15.63%     0.049      798.03   0.00000  0.00000  2988
cached-awb-on                 4000  4096    4  475.21 47.65%     0.092     1026.38   0.00000  0.00000   997
cached-awb-on                 4000  4096    8  465.94 65.01%     0.184     1639.50   0.00000  0.00000   717

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-on                 4000  4096    1    0.84 0.810%    13.934      115.33   0.00000  0.00000   104
cached-awb-on                 4000  4096    2    1.58 4.261%    14.398      116.67   0.00000  0.00000    37
cached-awb-on                 4000  4096    4    2.93 12.62%    14.281      110.91   0.00000  0.00000    23
cached-awb-on                 4000  4096    8    4.68 31.20%    17.261      253.25   0.00000  0.00000    15

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-on                 4000  4096    1  498.96 61.54%     0.023      545.80   0.00000  0.00000   811
cached-awb-on                 4000  4096    2  497.93 155.0%     0.046      600.66   0.00000  0.00000   321
cached-awb-on                 4000  4096    4  484.76 339.7%     0.094      625.07   0.00000  0.00000   143
cached-awb-on                 4000  4096    8  482.63 971.8%     0.185      512.20   0.00000  0.00000    50

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-on                 4000  4096    1   11.13 0.189%     0.009        0.05   0.00000  0.00000  5859
cached-awb-on                 4000  4096    2   10.71     0%     0.009        0.06   0.00000  0.00000     0
cached-awb-on                 4000  4096    4   10.66 1.092%     0.010        0.08   0.00000  0.00000   977
cached-awb-on                 4000  4096    8   10.21 11.76%     0.017        0.09   0.00000  0.00000    87

START TESTS IO = cached WC = awb PC = off

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-off                4000  4096    1  412.57 14.48%     0.028      784.63   0.00000  0.00000  2849
cached-awb-off                4000  4096    2  518.59 19.50%     0.045      695.77   0.00000  0.00000  2659
cached-awb-off                4000  4096    4  520.66 47.19%     0.083     1043.65   0.00000  0.00000  1103
cached-awb-off                4000  4096    8  519.11 85.76%     0.166     1363.41   0.00000  0.00000   605

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-off                4000  4096    1    0.82 0.674%    14.227      158.82   0.00000  0.00000   122
cached-awb-off                4000  4096    2    1.55 4.241%    14.736      157.93   0.00000  0.00000    37
cached-awb-off                4000  4096    4    2.95 12.04%    14.513      120.55   0.00000  0.00000    25
cached-awb-off                4000  4096    8    4.94 31.88%    17.059      137.71   0.00000  0.00000    16

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-off                4000  4096    1  488.89 61.01%     0.023      456.56   0.00000  0.00000   801
cached-awb-off                4000  4096    2  491.26 149.0%     0.046      512.93   0.00000  0.00000   330
cached-awb-off                4000  4096    4  485.30 328.9%     0.094      529.83   0.00000  0.00000   148
cached-awb-off                4000  4096    8  455.13 925.3%     0.197      562.44   0.00000  0.00000    49

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
cached-awb-off                4000  4096    1    9.74 0.083%     0.009        0.05   0.00000  0.00000 11719
cached-awb-off                4000  4096    2    8.70     0%     0.009        0.07   0.00000  0.00000     0
cached-awb-off                4000  4096    4    9.30 3.175%     0.010        0.09   0.00000  0.00000   293
cached-awb-off                4000  4096    8    8.25 9.569%     0.016        0.10   0.00000  0.00000    86

START TESTS IO = direct WC = wt PC = on

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-on                  4000  4096    1  470.31 13.90%     0.025      122.05   0.00000  0.00000  3382
direct-wt-on                  4000  4096    2  608.38 20.54%     0.038      230.56   0.00000  0.00000  2961
direct-wt-on                  4000  4096    4  593.23 79.81%     0.073      322.20   0.00000  0.00000   743
direct-wt-on                  4000  4096    8  602.60 171.7%     0.146      984.91   0.00000  0.00000   351

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-on                  4000  4096    1    0.79 0.695%    14.800      127.98   0.00000  0.00000   114
direct-wt-on                  4000  4096    2    1.41 3.943%    16.363      126.50   0.00000  0.00000    36
direct-wt-on                  4000  4096    4    2.72 11.50%    15.148      215.17   0.00000  0.00000    24
direct-wt-on                  4000  4096    8    4.62 38.64%    18.004      153.71   0.00000  0.00000    12

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-on                  4000  4096    1  458.64 57.13%     0.025      230.74   0.00000  0.00000   803
direct-wt-on                  4000  4096    2  290.16 94.70%     0.079      824.62   0.00000  0.00000   306
direct-wt-on                  4000  4096    4  229.99 148.4%     0.199      923.75   0.00000  0.00000   155
direct-wt-on                  4000  4096    8  208.63 313.7%     0.433     1068.90   0.00000  0.00000    66

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-on                  4000  4096    1    5.28 0.090%     0.009        0.06   0.00000  0.00000  5859
direct-wt-on                  4000  4096    2    6.08 0.103%     0.009        0.05   0.00000  0.00000  5859
direct-wt-on                  4000  4096    4    6.40 3.657%     0.010        0.08   0.00000  0.00000   175
direct-wt-on                  4000  4096    8    6.17 8.685%     0.017        0.15   0.00000  0.00000    71

START TESTS IO = direct WC = wt PC = off

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-off                 4000  4096    1  467.03 17.57%     0.025      132.30   0.00000  0.00000  2657
direct-wt-off                 4000  4096    2  611.99 34.98%     0.038      211.65   0.00000  0.00000  1749
direct-wt-off                 4000  4096    4  589.11 85.15%     0.075      356.54   0.00000  0.00000   692
direct-wt-off                 4000  4096    8  597.58 180.5%     0.147      892.34   0.00000  0.00000   331

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-off                 4000  4096    1    0.79 0.862%    14.831      100.28   0.00000  0.00000    92
direct-wt-off                 4000  4096    2    1.51 4.213%    15.191      115.55   0.00000  0.00000    36
direct-wt-off                 4000  4096    4    2.72 11.66%    15.295      106.87   0.00000  0.00000    23
direct-wt-off                 4000  4096    8    4.55 37.06%    18.315      142.48   0.00000  0.00000    12

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-off                 4000  4096    1  397.09 52.86%     0.029      239.47   0.00000  0.00000   751
direct-wt-off                 4000  4096    2  240.42 81.34%     0.095     1297.71   0.00000  0.00000   296
direct-wt-off                 4000  4096    4  181.09 126.6%     0.249     1506.76   0.00000  0.00000   143
direct-wt-off                 4000  4096    8  157.02 205.9%     0.584     1870.55   0.00000  0.00000    76

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-wt-off                 4000  4096    1    5.95 0.050%     0.010        0.06   0.00000  0.00000 11719
direct-wt-off                 4000  4096    2    6.00     0%     0.009        0.05   0.00000  0.00000     0
direct-wt-off                 4000  4096    4    5.79 1.582%     0.010        0.06   0.00000  0.00000   366
direct-wt-off                 4000  4096    8    5.94 32.05%     0.017        0.07   0.00000  0.00000    19

START TESTS IO = direct WC = awb PC = on

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-on                 4000  4096    1  409.87 16.53%     0.028      628.68   0.00000  0.00000  2479
direct-awb-on                 4000  4096    2  479.43 17.85%     0.048      546.33   0.00000  0.00000  2686
direct-awb-on                 4000  4096    4  469.68 48.44%     0.094     1023.92   0.00000  0.00000   970
direct-awb-on                 4000  4096    8  470.86 52.61%     0.185     1559.41   0.00000  0.00000   895

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-on                 4000  4096    1    0.81 0.773%    14.482      143.94   0.00000  0.00000   105
direct-awb-on                 4000  4096    2    1.60 4.115%    14.414       96.66   0.00000  0.00000    39
direct-awb-on                 4000  4096    4    2.95 13.23%    14.302       90.20   0.00000  0.00000    22
direct-awb-on                 4000  4096    8    4.91 26.20%    17.121      146.82   0.00000  0.00000    19

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-on                 4000  4096    1  496.21 61.84%     0.023       51.54   0.00000  0.00000   802
direct-awb-on                 4000  4096    2  500.74 151.4%     0.046      346.99   0.00000  0.00000   331
direct-awb-on                 4000  4096    4  493.30 326.0%     0.092      367.83   0.00000  0.00000   151
direct-awb-on                 4000  4096    8  482.48 972.3%     0.185      394.22   0.00000  0.00000    50

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-on                 4000  4096    1   12.62 0.107%     0.009        0.07   0.00000  0.00000 11719
direct-awb-on                 4000  4096    2   12.54 0.213%     0.009        0.06   0.00000  0.00000  5859
direct-awb-on                 4000  4096    4   11.28 3.466%     0.010        0.07   0.00000  0.00000   325
direct-awb-on                 4000  4096    8    9.51 4.543%     0.017        0.09   0.00000  0.00000   209

START TESTS IO = direct WC = awb PC = off

Sequential Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-off                4000  4096    1  419.91 14.83%     0.028      836.43   0.00000  0.00000  2830
direct-awb-off                4000  4096    2  508.28 20.26%     0.045      741.52   0.00000  0.00000  2508
direct-awb-off                4000  4096    4  515.65 57.17%     0.087     1018.63   0.00000  0.00000   902
direct-awb-off                4000  4096    8  519.17 90.11%     0.168     1168.69   0.00000  0.00000   576

Random Reads
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-off                4000  4096    1    0.83 0.620%    14.187      122.07   0.00000  0.00000   133
direct-awb-off                4000  4096    2    1.57 3.062%    14.386      145.25   0.00000  0.00000    51
direct-awb-off                4000  4096    4    2.93 12.51%    14.586      132.94   0.00000  0.00000    23
direct-awb-off                4000  4096    8    4.81 31.41%    17.062      138.90   0.00000  0.00000    15

Sequential Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-off                4000  4096    1  489.41 64.14%     0.023       98.94   0.00000  0.00000   763
direct-awb-off                4000  4096    2  475.58 145.4%     0.048      173.13   0.00000  0.00000   327
direct-awb-off                4000  4096    4  463.76 301.8%     0.098      239.71   0.00000  0.00000   154
direct-awb-off                4000  4096    8  440.92 875.4%     0.205      537.00   0.00000  0.00000    50

Random Writes
                              File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
direct-awb-off                4000  4096    1    9.07     0%     0.010        0.07   0.00000  0.00000     0
direct-awb-off                4000  4096    2    9.85 0.168%     0.009        0.06   0.00000  0.00000  5859
direct-awb-off                4000  4096    4    9.51 3.894%     0.010        0.09   0.00000  0.00000   244
direct-awb-off                4000  4096    8    8.65 6.868%     0.018        0.11   0.00000  0.00000   126

Skripts

#!/bin/bash
storcli=/opt/MegaRAID/storcli/storcli64

$storcli /c0/vall set iopolicy=cached
$storcli /c0/vall set wrcache=awb
$storcli /c0/vall set pdcache=on

LOGFILE=/root/tio.md.txt

for iopolicy  in cached  direct
do
 $storcli /c0/vall set iopolicy=$iopolicy
        IO=$iopolicy 
        echo = IO=$IO

 for wrcache in wt awb
 do
  $storcli /c0/vall set wrcache=$wrcache
  WC=$wrcache
  echo == WC=$WC
  for pdcache in on off
  do
   $storcli /c0/vall set pdcache=$pdcache
   PC=$pdcache
   echo === PC=$PC
   echo 
   IDENT=$IO-$WC-$PC
   $storcli /c0/vall show | grep RAID | tee -a $LOGFILE
   echo  ==== START TESTS IO = $IO    WC = $WC    PC = $PC    =====  `date +%T` ==== | tee -a $LOGFILE
   sleep 3 
   tiobench --size 4000 --ident $IDENT --dir=/mnt/test/ --numruns 3 | tee -a $LOGFILE
  done
 done
done

Testējam RAID 1+0 pret RAID 10 

RAID 1+0 4 diski

[root@bfsa ~]# cat /proc/mdstat 
Personalities : [raid10] [raid1] [raid0] 
md3 : active raid0 md2[1] md1[0]
      83841024 blocks super 1.2 512k chunks
md2 : active raid1 sdj1[1] sdd1[0]
      41920832 blocks super 1.2 [2/2] [UU]     
md1 : active raid1 sdc1[1] sdb1[0]
      41920832 blocks super 1.2 [2/2] [UU]

tiobench --size 8000 --dir /mnt/test1+0/  | tee -a /root/tio.md1+0
================
Sequential Reads
2.6.32-358.2.1.el6.x86_64     8000  4096    1  188.48 16.65%     0.021       79.78   0.00000  0.00000  1132
2.6.32-358.2.1.el6.x86_64     8000  4096    2  297.19 52.23%     0.026       81.99   0.00000  0.00000   569
2.6.32-358.2.1.el6.x86_64     8000  4096    4  246.01 74.94%     0.055      448.62   0.00000  0.00000   328
2.6.32-358.2.1.el6.x86_64     8000  4096    8  205.31 138.7%     0.146      304.43   0.00000  0.00000   148

Random Reads
2.6.32-358.2.1.el6.x86_64     8000  4096    1    0.73 0.354%     5.357       47.57   0.00000  0.00000   206
2.6.32-358.2.1.el6.x86_64     8000  4096    2    1.48 1.384%     5.134       41.11   0.00000  0.00000   107
2.6.32-358.2.1.el6.x86_64     8000  4096    4    1.89 3.478%     6.232       58.14   0.00000  0.00000    54
2.6.32-358.2.1.el6.x86_64     8000  4096    8    3.11 12.86%     8.934       99.02   0.00000  0.00000    24

Sequential Writes
2.6.32-358.2.1.el6.x86_64     8000  4096    1  197.38 40.46%     0.017      102.73   0.00000  0.00000   488
2.6.32-358.2.1.el6.x86_64     8000  4096    2  188.81 87.68%     0.036     2897.20   0.00005  0.00000   215
2.6.32-358.2.1.el6.x86_64     8000  4096    4  144.03 161.3%     0.095     5853.08   0.00059  0.00000    89
2.6.32-358.2.1.el6.x86_64     8000  4096    8  120.28 358.7%     0.229    10946.50   0.00171  0.00015    34

Random Writes
2.6.32-358.2.1.el6.x86_64     8000  4096    1    2.81 1.688%     0.006        0.05   0.00000  0.00000   166
2.6.32-358.2.1.el6.x86_64     8000  4096    2    2.81 3.738%     0.007        0.05   0.00000  0.00000    75
2.6.32-358.2.1.el6.x86_64     8000  4096    4    2.68 9.202%     0.009        0.05   0.00000  0.00000    29
2.6.32-358.2.1.el6.x86_64     8000  4096    8    2.73 18.87%     0.012        4.92   0.00000  0.00000    14

RAID 10 4 diski

mdadm --create --verbose /dev/md10 --level=10 --raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sdj1

[root@bfsa ~]# cat /proc/mdstat 
Personalities : [raid10] [raid1] [raid0] 
md10 : active raid10 sdj1[3] sdd1[2] sdc1[1] sdb1[0]
      83841024 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      
mkfs.ext4 /dev/md10
mount /dev/md10 /mnt/test10
      
[root@bfsa ~]# tiobench --size 8000 --dir /mnt/test10  | tee -a /root/tio.md10
Run #1: /usr/bin/tiotest -t 8 -f 1000 -r 500 -b 4096 -d /mnt/test10 -TT

====
Sequential Reads
2.6.32-358.2.1.el6.x86_64     8000  4096    1  115.54 9.296%     0.034       49.51   0.00000  0.00000  1243
2.6.32-358.2.1.el6.x86_64     8000  4096    2  217.76 36.41%     0.036      111.80   0.00000  0.00000   598
2.6.32-358.2.1.el6.x86_64     8000  4096    4  167.97 55.71%     0.092      187.27   0.00000  0.00000   302
2.6.32-358.2.1.el6.x86_64     8000  4096    8  164.52 103.0%     0.176      297.66   0.00000  0.00000   160

Random Reads
2.6.32-358.2.1.el6.x86_64     8000  4096    1    0.76 0.338%     5.164       34.67   0.00000  0.00000   223
2.6.32-358.2.1.el6.x86_64     8000  4096    2    1.54 1.263%     5.027       19.42   0.00000  0.00000   122
2.6.32-358.2.1.el6.x86_64     8000  4096    4    2.38 4.458%     6.453       76.78   0.00000  0.00000    53
2.6.32-358.2.1.el6.x86_64     8000  4096    8    2.77 10.43%     9.030      116.49   0.00000  0.00000    27

Sequential Writes
2.6.32-358.2.1.el6.x86_64     8000  4096    1  198.69 39.37%     0.017      126.57   0.00000  0.00000   505
2.6.32-358.2.1.el6.x86_64     8000  4096    2  140.85 73.33%     0.050     3834.55   0.00039  0.00000   192
2.6.32-358.2.1.el6.x86_64     8000  4096    4  121.15 169.8%     0.114     6588.93   0.00103  0.00000    71
2.6.32-358.2.1.el6.x86_64     8000  4096    8  105.19 501.7%     0.256    11103.32   0.00249  0.00005    21

Random Writes
2.6.32-358.2.1.el6.x86_64     8000  4096    1    2.86 1.776%     0.006        0.04   0.00000  0.00000   161
2.6.32-358.2.1.el6.x86_64     8000  4096    2    2.79 4.039%     0.007        0.06   0.00000  0.00000    69
2.6.32-358.2.1.el6.x86_64     8000  4096    4    2.65 11.11%     0.009        0.05   0.00000  0.00000    24
2.6.32-358.2.1.el6.x86_64     8000  4096    8    2.67 32.15%     0.012        8.04   0.00000  0.00000     8

RAID 1+0 6 diski

mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1 
mdadm --create --verbose /dev/md2 --level=1 --raid-devices=2 /dev/sdd1 /dev/sdj1 
mdadm --create --verbose /dev/md3 --level=1 --raid-devices=2 /dev/sdk1 /dev/sdn1 
mdadm --create --verbose /dev/md4 --level=stripe --raid-devices=3 /dev/md1 /dev/md2 /dev/md3

mkfs.ext4 /dev/md4
mount /dev/md4 /mnt/test1+0/

[root@bfsa ~]# cat /proc/mdstat 
Personalities : [raid10] [raid1] [raid0] 
md4 : active raid0 md3[2] md2[1] md1[0]
      90173952 blocks super 1.2 512k chunks      
md3 : active raid1 sdn1[1] sdk1[0]
      6333440 blocks super 1.2 [2/2] [UU]      
md2 : active raid1 sdj1[1] sdd1[0]
      41920832 blocks super 1.2 [2/2] [UU]      
md1 : active raid1 sdc1[1] sdb1[0]
      41920832 blocks super 1.2 [2/2] [UU]

[root@bfsa ~]# tiobench --size 8000 --dir /mnt/test1+0/  | tee -a /root/tio.md1+06drv
Run #1: /usr/bin/tiotest -t 8 -f 1000 -r 500 -b 4096 -d /mnt/test1+0/ -TT

================
Sequential Reads
2.6.32-358.2.1.el6.x86_64     8000  4096    1  277.03 23.62%     0.014       48.44   0.00000  0.00000  1173
2.6.32-358.2.1.el6.x86_64     8000  4096    2  392.34 70.15%     0.020      145.79   0.00000  0.00000   559
2.6.32-358.2.1.el6.x86_64     8000  4096    4  266.50 83.11%     0.050      159.51   0.00000  0.00000   321
2.6.32-358.2.1.el6.x86_64     8000  4096    8  220.63 145.7%     0.131      338.48   0.00000  0.00000   151

Random Reads
2.6.32-358.2.1.el6.x86_64     8000  4096    1    0.80 0.402%     4.903       17.29   0.00000  0.00000   198
2.6.32-358.2.1.el6.x86_64     8000  4096    2    1.54 1.694%     5.057       27.96   0.00000  0.00000    91
2.6.32-358.2.1.el6.x86_64     8000  4096    4    1.98 3.818%     5.796       44.83   0.00000  0.00000    52
2.6.32-358.2.1.el6.x86_64     8000  4096    8    2.72 11.67%     9.141      127.06   0.00000  0.00000    23

Sequential Writes
2.6.32-358.2.1.el6.x86_64     8000  4096    1  280.48 60.28%     0.012       85.91   0.00000  0.00000   465
2.6.32-358.2.1.el6.x86_64     8000  4096    2  266.83 131.8%     0.025     1967.06   0.00000  0.00000   202
2.6.32-358.2.1.el6.x86_64     8000  4096    4  147.23 173.3%     0.093     5948.60   0.00054  0.00000    85
2.6.32-358.2.1.el6.x86_64     8000  4096    8  136.94 345.4%     0.205     6689.59   0.00103  0.00000    40

Random Writes
2.6.32-358.2.1.el6.x86_64     8000  4096    1    4.27 2.488%     0.006        0.04   0.00000  0.00000   172
2.6.32-358.2.1.el6.x86_64     8000  4096    2    4.31 5.630%     0.007        0.06   0.00000  0.00000    77
2.6.32-358.2.1.el6.x86_64     8000  4096    4    2.86 9.743%     0.009        0.06   0.00000  0.00000    29
2.6.32-358.2.1.el6.x86_64     8000  4096    8    2.57 18.06%     0.012        5.15   0.00000  0.00000    14

RAID 10 6 diski

mdadm --create --verbose /dev/md10 --level=10 --raid-devices=6 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sdj1 /dev/sdk1 /dev/sdn1

[root@bfsa ~]# cat /proc/mdstat 
Personalities : [raid10] [raid1] [raid0] 
md10 : active raid10 sdn1[5] sdk1[4] sdj1[3] sdd1[2] sdc1[1] sdb1[0]
      19000320 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]

mkfs.ext4 /dev/md10
mount /dev/md10 /mnt/test10

[root@bfsa ~]# tiobench --size 8000 --dir /mnt/test10  | tee -a /root/tio.md106drv
Run #1: /usr/bin/tiotest -t 8 -f 1000 -r 500 -b 4096 -d /mnt/test10 -TT

================
Sequential Reads
2.6.32-358.2.1.el6.x86_64     8000  4096    1  166.17 13.41%     0.023       36.49   0.00000  0.00000  1239
2.6.32-358.2.1.el6.x86_64     8000  4096    2  251.41 42.26%     0.031       87.61   0.00000  0.00000   595
2.6.32-358.2.1.el6.x86_64     8000  4096    4  214.33 71.29%     0.072      206.46   0.00000  0.00000   301
2.6.32-358.2.1.el6.x86_64     8000  4096    8  218.76 148.0%     0.141      326.60   0.00000  0.00000   148

Random Reads
2.6.32-358.2.1.el6.x86_64     8000  4096    1    0.78 0.307%     5.033       60.76   0.00000  0.00000   252
2.6.32-358.2.1.el6.x86_64     8000  4096    2    1.45 1.176%     5.183      307.60   0.00000  0.00000   123
2.6.32-358.2.1.el6.x86_64     8000  4096    4    2.42 4.206%     5.972       44.55   0.00000  0.00000    57
2.6.32-358.2.1.el6.x86_64     8000  4096    8    4.10 16.05%     7.377       65.87   0.00000  0.00000    26

Sequential Writes
2.6.32-358.2.1.el6.x86_64     8000  4096    1  273.81 58.18%     0.013       63.96   0.00000  0.00000   471
2.6.32-358.2.1.el6.x86_64     8000  4096    2  160.74 91.67%     0.044     2419.02   0.00063  0.00000   175
2.6.32-358.2.1.el6.x86_64     8000  4096    4  133.57 223.6%     0.108     5177.93   0.00156  0.00000    60
2.6.32-358.2.1.el6.x86_64     8000  4096    8  159.02 646.6%     0.169     7223.63   0.00137  0.00000    25

Random Writes
2.6.32-358.2.1.el6.x86_64     8000  4096    1    4.19 2.684%     0.006        0.03   0.00000  0.00000   156
2.6.32-358.2.1.el6.x86_64     8000  4096    2    4.20 6.282%     0.007        0.06   0.00000  0.00000    67
2.6.32-358.2.1.el6.x86_64     8000  4096    4    4.05 17.82%     0.009        0.08   0.00000  0.00000    23
2.6.32-358.2.1.el6.x86_64     8000  4096    8    3.92 47.87%     0.011        4.62   0.00000  0.00000     8

Intel HW Raid testi

Serveris:
HW RAID kontrolieris:

• HW RAID5 sējums ar 4 diskiem: (4-1) x 500 GB
• HW RAID5 sējums ar 8 diskiem: (8-1) x 500 GB

Kontroliera uzstādījumi WRITE THROUGH

Cache Policy:WriteThrough, ReadAheadNone, Direct, No Write Cache if bad BBU

iozone 8 disku RAIDAM:

KB       reclen   write rewrite    read    reread   rread  rwrite
8388608    2048   47370   45865   201040   178148  103822  106027

iozone 4 disku RAIDAM:

KB       reclen   write rewrite    read    reread   rread  rwrite
6777216    2048   36553   36239   160195   162528

Kontroliera uzstādījumi WRITE BACK

Cache Policy:WriteBack, ReadAheadNone, Cached, Write Cache OK if bad BBU

iozone 8 disku RAIDAM:

KB        reclen   write rewrite    read    reread   rread  rwrite
33554432    2048  228497  177099   156933   156547   56818   92855
33554432    4096  224270  174636   153236   152991   79173  123178
33554432    8192  221834  175551   153467   153147  107403  139591
33554432   16384  223685  172882   152044   151842  130756  155928

iozone 4 disku RAIDAM:

KB        reclen   write rewrite    read    reread   rread  rwrite
33554432    2048   98454   84888   106007   106103   59516   56044
33554432    4096   99630   83457   104566   104922   74436   66626
33554432    8192   99731   83149   103485   104096   90022   74085
33554432   16384   99962   83023   103984   103552  101068   79891

Ubuntu spoguļa kopēšana ar rsync

no 4 disku sējuma uz 8 disku sējumu:
38673623.15 bytes/sec
( 38,7 MB/s )

no 8 disku sējuma uz 4 disku sējumu:
46673383.85 bytes/sec
( 46,7 MB/s )

RAID veiktspējas

Apkopošu šeit dažu man pieejamo serveru cieto disku veiktspējas programmatūras RAID'os n

Līdz šim nav sanācis iegūt labu veiktspēju RAID veiktspēju ar hardware RAID'iem un tos apsteidz progammatūras RAID'i.

Šis saraksts nekādi nepretendē uz kvalitatīvu pētījumu, bet ir domāts kā ieskats dažādu programmatūras RAID'u savstarpējā salīdzinājumā.

Veiktspēja tiek mērīta ar iozone , testa faila lielums = 2x RAM atmiņas lielums.

Serveri pamatā strādā uz Debian OS. RAID'i tiek veidoti ar noklusētajiem parametriem ar komandu mdadm. Visi sējumi tiek formatēti ar ext3 failu sistēmu.

Iozone komanda ir:

iozone -i0 -i1 -c -e -a -n 1g -g 16g -y 2048 -s 2048 -q 8192 -f /media/raidX/tmpfile

Serveris 1.

RAM: 8GB
Procesors: 2 x Intel Xeon 2,66 GHz

SCSI storage controller: LSI Logic / Symbios Logic SAS1064E PCI-Express Fusion-MPT SAS (rev 02)
3 x 73GB SAS diski (FUJITSU Model: MAX3073RC)

Rakstīšanas ātrums uz sākuma sējuma - 65MB, lasīšana ~ 80MB

RAID'iem izmantotie sējumi ir 60GB lieli un atrodas

RAID5

Programmatūras RAID5 (stripe set with parity) no 3 sējumiem;

              KB  reclen   write rewrite    read    reread
        16777216    2048   64290   62756   127831   145092                                                                          
        16777216    4096   60875   63600   149913   128595                                                                          
        16777216    8192   60329   65239   136654   149085 

jeb rakstīšana ~ 62-63 MB/s, lasīšana ~ 139-140 MB/s
hmmm... Vajadzētu būt lielākam rakstīšanas ātrumam, arī nolasīšana varētu būt ātrāka

RAID0

Programmatūras RAID0 (stripe set) no 3 sējumiem:

              KB  reclen   write rewrite    read    reread
        16777216    2048  154394  128102   249265   200720                                                                          
        16777216    4096  163496  138319   194408   225419                                                                          
        16777216    8192  166613  117622   226446   247582

jeb rakstīšana ~128-161MB un lasīšana 239-240MB. Rezultāti apmierinoši, bet ir novērojams ļoti straujš veiktspējas samazinājums pie pārrakstīšanas (rewrite). Jāatzīmē, ka rakstīšanas un pārrakstīšanas ātrums pie faila izmēra, kas vienāds ar operatīvo atmiņas lielumu, ir ~ 175MB/s (un pie mazākiem testa faila izmēriem pārrakstīšana, protams, ir ātrāka un ir ~ 200MB/s)

Serveris 2.

RAM: 16 GB
Procesors: 2 x IntelXeon X5355 (2.66GHz)
SCSI storage controller: LSI Logic / Symbios Logic SAS1064E PCI-Express Fusion-MPT SAS (rev 04)
4 x 146GB 15k SAS diski (FUJITSU Model: MAX3147RC)

Rakstīšanas ātrums uz sākuma sējuma - ???MB, lasīšana ~ ???MB (ja nemaldos ~70-80MB)

RAID'iem izmantotie sējumi ir pa visu disku.

RAID0

Programmatūras RAID0 no 4 sējumiem
Sējums = 4 x 147 = 588 GB liels

              KB  reclen   write rewrite    read    reread 
        33554432    2048  272929  246462   356589   360505                                                                          
        33554432    4096  272003  239118   359652   360772                                                                          
        33554432    8192  273092  224971   360500   360682

jeb rakstīšanas ātrums ~ 237-272 MB/s, lasīšanas - 359-360 MB/s - rezultāti teicami - maksimālā veiktspēja, diemžēl bez bojājumpiecietības.

RAID1

Programmatūras RAID1 (spogulis) no 2 sējumiem
no 4 sējumiem var uzbūvēt 2 spoguļus, katru 147 GB lielu

              KB  reclen   write rewrite    read    reread 
        33554432    2048   63560   66897    89109    89108                                                                          
        33554432    4096   63152   67135    89091    89098                                                                          
        33554432    8192   64583   66674    89119    89084 

jeb rakstīšanas ātrums ~ 64-67 MB/s, lasīšanas - 89 MB/s - rezultāti ir līdzīgi ar vienkārša sējuma veiktspēju.

RAID10

Programmatūras RAID10 ar 4 sējumiem
Apjoms 2 x 147 ~ 294 GB

              KB  reclen   write rewrite    read    reread 
        33554432    2048  128738  132318   179708   179775                                                                          
        33554432    4096  130096  130713   179728   179607                                                                          
        33554432    8192  128415  134781   179840   179632

jeb rakstīšanas ātrums ~ 129-132 MB/s, lasīšanas - 179 MB/s - veiktspēja ir 2x ātrāka, kā RAID1 un 2x mazāka kā RAID0 - ļoti labi mērogojas.

RAID5

Programmatūras RAID5 ar 4 sējumiem
Apjoms 147 x 3 ~ 441 GB

              KB  reclen   write rewrite    read    reread 
        33554432    2048   89080   89923   267805   268254                                                                          
        33554432    4096   88898   90062   268016   268378                                                                          
        33554432    8192   88224   90481   268047   268097

jeb rakstīšanas ātrums ~ 89-90 MB/s, lasīšanas - 268 MB/s - pieticīgs rakstīšanas ātrums, labs lasīšanas ātrums.

Serveris 3

Hardware RAID5
Kontrolieris - 3ware 9500 (bez BBU - write trough policy)
Diski - 6 x 250 GB SATA

              KB  reclen   write rewrite    read    reread
        16777216    2048   30977   29960    25984    24826
        16777216    4096   28402   29729    25695    25014
        16777216    8192   30090   28541    25465    24510

jeb rakstīšanas ātrums - 29MB, lasīšana - 25MB :(