title: Reading large files on Linux vs Windows (mmap vs CreateFileMapping/MapViewOfFile)
body: <p>I have to read some data line by line from a large file (more than 7GB), it contains a list of vertex coordinates and face to vertex connectivity information to form a mesh. I am also learning how to use <code>open</code>, <code>mmap</code> on Linux and <code>CreateFileA</code>, <code>CreateFileMapping</code>, <code>MapViewOfFile</code> on Windows. Both Linux and Windows versions are 64bit compiled.</p>
  <p>When I am on Linux (using docker) with <code>g++-10 test.cpp -O3 -std=c++17</code> I get around 6s.
  When I am on Windows (my actual PC) both with (version 19.29.30037 x64) <code>cl test.cpp /EHsc /O3 /std:c++17</code> I get 13s, and with <code>clang++-11</code> (from Visual Studio Build Tools) I get 11s.</p>
  <p>Both systems (same PC, but one is using docker) use the same exact code except for generating the <code>const char*</code> that represents the memory array and the <code>uint64_t</code> size that reprents the memory size.</p>
  <p>This is the way I switch platforms:</p>
  <pre><code>// includes for using a specific platform API
  #ifdef _WIN32
  #define WIN32_LEAN_AND_MEAN
  #include &lt;windows.h&gt;
  // using windows handle void*
  #define handle_type HANDLE
  #else
  #include &lt;fcntl.h&gt;
  #include &lt;sys/mman.h&gt;
  #include &lt;sys/stat.h&gt;
  #include &lt;unistd.h&gt;
  // using file descriptors
  #define handle_type int
  #endif
  </code></pre>
  <p>Specifically the code for getting the memory in an array of char-s is:</p>
  <pre><code>using uint_t = std::size_t;
  
  // open the file -----------------------------------------------------------------------------
  handle_type open(const std::string&amp; filename) {
  #ifdef _WIN32
    // see windows file mapping api for parameter explanation
    return ::CreateFileA(filename.c_str(), GENERIC_READ, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL); // private access
  #else
    return ::open(filename.c_str(), O_RDONLY);
  #endif
  }
  
  
  // get the memory size to later have a bound for reading -------------------------------------
  uint_t memory_size(handle_type fid) {
  #ifdef _WIN32
    LARGE_INTEGER size{};
    if (!::GetFileSizeEx(fid, &amp;size)) {
      std::cerr &lt;&lt; &quot;file not found\n&quot;;
      return size.QuadPart;
    }
    return size.QuadPart;
  #else
    struct stat sb;
    // get the file stats and check if not zero size
    if (fstat(fid, &amp;sb)) {
      std::cerr &lt;&lt; &quot;file not found\n&quot;;
      return decltype(sb.st_size){};
    }
    return sb.st_size;
  #endif
  }
  
  // get the actual char array to access memory ------------------------------------------------
  const char* memory_map(handle_type fid, uint_t memory_size) {
  #ifdef _WIN32
    HANDLE mapper = ::CreateFileMapping(fid, NULL, PAGE_READONLY, 0, 0, NULL);
    return reinterpret_cast&lt;const char*&gt;(::MapViewOfFile(mapper, FILE_MAP_READ, 0, 0, memory_size));
  #else
    return reinterpret_cast&lt;const char*&gt;(::mmap(NULL, memory_size, PROT_READ, MAP_PRIVATE, fid, 0));
  #endif
  }
  </code></pre>
  <p>I am completely new to this sort of parsing and was wondering if I am doing something wrong in choosing the parameters in the Windows API (to mimic the behaviour of mmap) or if the difference in time is a matter of compilers/systems and have to accept it?</p>
  <p>The actual time to open, get the memory size, and the memory map is negligible both on Linux and on Windows, the rest of the code is identical, as it only operates using the <code>const char*</code> and <code>size_t</code> info.</p>
  <p>Thanks for taking the time to read. Any tip is greatly appreciated and sorry if anything is unclear.</p>