title: SSD raw I/O benchmarks with random read/write
body: <p>My laptop has a SSD disk that has 512 byte physical disk sector size and 4,096 byte logical disk sector size. I'm working on an ACID database system that <em>has</em> to bypass all OS caches, so I write directly from allocated internal memory (RAM) to the SSD disk. I also extend the files <em>before</em> I run the tests and don't resize it during the tests. </p>
  
  <p>Now here is my  problem, according to <a href="http://www.tomshardware.com/charts/ssd-charts-2013/benchmarks,129.html" rel="nofollow">SSD benchmarks</a> random read &amp; write should be in the range 30 MB/s to 90 MB/s, respectively. But here is my (rather horrible) telemetry from my numerous perfrmance tests:</p>
  
  <ul>
  <li>1.2 MB/s when reading random 512 byte blocks (physical sector size) </li>
  <li>512 KB/s when writing random 512 byte blocks (physical sector size)</li>
  <li>8.5 MB/s when reading random 4,096 byte blocks (logical sector size)</li>
  <li>4.9 MB/s when writing random 4,096 byte blocks (logical sector size)</li>
  </ul>
  
  <p>In addition to using asynchronous I/O I also set the <code>FILE_SHARE_READ</code> and <code>FILE_SHARE_WRITE</code> flags to disable all OS buffering - because our database is ACID I must do this, I also tried <code>FlushFileBuffers()</code> but that gave me even worse performance. I also wait for each async I/O operation to complete as is required by some of our code.</p>
  
  <p>Here is my code, is there are problem with it or am I stuck with this bad I/O performance?</p>
  
  <pre><code>HANDLE OpenFile(const wchar_t *fileName)
  {
      // Set access method
      DWORD desiredAccess = GENERIC_READ | GENERIC_WRITE ;
  
      // Set file flags
      DWORD fileFlags = FILE_FLAG_WRITE_THROUGH | FILE_FLAG_NO_BUFFERING /*| FILE_FLAG_RANDOM_ACCESS*/;
  
      //File or device is being opened or created for asynchronous I/O
      fileFlags |= FILE_FLAG_OVERLAPPED ;
  
      // Exlusive use (no share mode)
      DWORD shareMode = 0;
  
      HANDLE hOutputFile = CreateFile(
          // File name
          fileName,
          // Requested access to the file 
          desiredAccess,
          // Share mode. 0 equals exclusive lock by the process
          shareMode,
          // Pointer to a security attribute structure
          NULL,
          // Action to take on file
          CREATE_NEW,
          // File attributes and flags
          fileFlags,
          // Template file
          NULL
      );
      if (hOutputFile == INVALID_HANDLE_VALUE)
      {
          int lastError = GetLastError();
          std::cerr &lt;&lt; "Unable to create the file '" &lt;&lt; fileName &lt;&lt; "'. [CreateFile] error #" &lt;&lt; lastError &lt;&lt; "." &lt;&lt; std::endl;
      }
  
      return hOutputFile;
  }
  
  DWORD ReadFromFile(HANDLE hFile, void *outData, _UINT64 bytesToRead, _UINT64 location, OVERLAPPED *overlappedPtr, 
      asyncIoCompletionRoutine_t completionRoutine)
  {
      DWORD bytesRead = 0;
  
      if (overlappedPtr)
      {
          // Windows demand that you split the file byte locttion into high &amp; low 32-bit addresses
          overlappedPtr-&gt;Offset = (DWORD)_UINT64LO(location);
          overlappedPtr-&gt;OffsetHigh = (DWORD)_UINT64HI(location);
  
          // Should we use a callback function or a manual event
          if (!completionRoutine &amp;&amp; !overlappedPtr-&gt;hEvent)
          {
              // No manual event supplied, so create one. The caller must reset and close it themselves
              overlappedPtr-&gt;hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
              if (!overlappedPtr-&gt;hEvent)
              {
                  DWORD errNumber = GetLastError();
                  std::wcerr &lt;&lt; L"Could not create a new event. [CreateEvent] error #" &lt;&lt; errNumber &lt;&lt; L".";
              }
          }
      }
  
      BOOL result = completionRoutine ? 
          ReadFileEx(hFile, outData, (DWORD)(bytesToRead), overlappedPtr, completionRoutine) : 
          ReadFile(hFile, outData, (DWORD)(bytesToRead), &amp;bytesRead, overlappedPtr);
  
      if (result == FALSE)
      {
          DWORD errorCode = GetLastError();
          if (errorCode != ERROR_IO_PENDING)
          {
              std::wcerr &lt;&lt; L"Can't read sectors from file. [ReadFile] error #" &lt;&lt; errorCode &lt;&lt; L".";
          }
      }
  
      return bytesRead;
  }
  </code></pre>