SSD Block Device Performance Tuning (Linux)

When I'm developing a storage system using SSD as the media,  I encounter quite a few performance issues and here is my personal note.

Read ahead

My system uses mmap() to map a disk file into virtual memory, then directly read/write to the virtual memory space.  I use nvme SSD.

When I run sequential read across the file I got great performance.  However if I run random 4KB read the performance drops significantly.

SSD default setting on Ubuntu 16 is:

hcd@hcd8:~$ sudo blockdev --report /dev/nvme1n1

RO    RA   SSZ   BSZ   StartSec            Size   Device

rw    256   512   512          0    960197124096   /dev/nvme1n1

Here the readahead is 256 sectors way too much.  I change to 16 sectors:

hcd@hcd8:~$ sudo blockdev --setra 16 /dev/nvme0n1

Boom!  Now random read performance spikes by an order of magnitude.