Testing Cluster Redo Log Write Performance with dd

• Note:  This is a destructive test  !
• Don’t run this test in a production env or at least use newly created disk without any  DATA !

• Use bs=128k for writing 128k blocks
• Use oflag=sync to bypass buffer cache ( Oracle uses same flag )

Test Instances one after the other to get an idea about your single instance performance ( run test 3x )

Instance grac41:
# dd if=/dev/zero  of=/dev/asmdisk1_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 61.1947 s, 10.7 MB/s
--> Instance grac41:  81,96 IOPs   Transfer Rate:  10.7 MB/s

Instance grac42:
# dd if=/dev/zero  of=/dev/asmdisk2_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 41.4141 s, 15.8 MB/s
--> Instance grac42:  121.95 IOPs   Transfer Rate:  15.8 MB/s

Instance grac43:
#   dd if=/dev/zero  of=/dev/asmdisk3_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 62.8516 s, 10.4 MB/s
--> Instance grac43:  80,64 IOPs   Transfer Rate:  10.4 MB/s

• We expect a cummulated Write I/O rate from about 280 IOPs for our whole cluster !

Testing all instances by running dd command in parallel to figure out any bottlenecks

[root@grac41 dev]# dd if=/dev/zero  of=/dev/asmdisk1_test bs=128k oflag=sync count=5000 
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 97.604 s, 6.7 MB/s
[root@grac42 ~]# dd if=/dev/zero  of=/dev/asmdisk2_test bs=128k oflag=sync count=5000
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 97.645 s, 6.7 MB/s
[root@grac43 ~]# dd if=/dev/zero  of=/dev/asmdisk3_test bs=128k oflag=sync count=5000 
5000+0 records in
5000+0 records out
655360000 bytes (655 MB) copied, 118.749 s, 5.5 MB/s
--> Clusterwide  IOPs:  15.000 / 118 = 127 IOPs

• IOPs drops from expected rate of 280 IOPs to 127 IOPs
• There is a Controller and/or Disk bottleneck and needs further review from storage group
• If suffering “LOG FILE SYNC” Wait  event IOPs sequential write performance is the limiting factor and you should review whether your hardware is scaling well

Reference

• http://www.oaktable.net/content/lgwr-redo-write-times-log-file-parallel-write