title: When to do or not do INVLPG, MOV to CR3 to minimize TLB flushing
body: <h2>Prologue</h2>
  
  <p>I am an operating system hobbyist, and my kernel runs on 80486+, and already supports virtual memory.</p>
  
  <p>Starting from 80386, the x86 processor family by Intel and various clones thereof has supported virtual memory with paging. It is well known that when the <code>PG</code> bit in <code>CR0</code> is set, the processor uses virtual address translation. Then, the <code>CR3</code> register points to the top-level page directory, that is the root for 2-4 levels of page table structures that map the virtual addresses to physical addresses.</p>
  
  <p>The processor does not consult these tables for each virtual address generated, instead caching them in a structure called <a href="http://en.wikipedia.org/wiki/Translation_lookaside_buffer" rel="noreferrer">Translation Lookaside Buffer</a>, or TLB. However, when changes to the page tables are made, the TLB needs to be flushed. On 80386 processors, this flush would be done by 
  reloading (<code>MOV</code>) <code>CR3</code> with the top level page directory address, or a task switch. This supposedly unconditionally flushes all the TLB entries. As I understand, it would be perfectly valid for a virtual memory system to <strong>always reload CR3</strong> after <strong>any</strong> change.</p>
  
  <p>This is wasteful, since the TLB would now throw out completely good entries, thus in 80486 processors the <a href="http://faydoc.tripod.com/cpu/invlpg.htm" rel="noreferrer"><code>INVLPG</code></a> instruction was introduced. <code>INVLPG</code> will invalidate the TLB entry matching the source operand address. </p>
  
  <p>Yet starting with Pentium Pro, we also have global pages that are not flushed with the moves to <code>CR3</code> or task switch; and AMD x86-64 ISA says that some upper level page table structures might be cached and not invalidated by <code>INVLPG</code>. To get a coherent picture of what is needed and what is not needed on each ISA one would really need to download a 1000-page datasheet for a multitudes of ISAs released since 80s to read a couple pages therein, and even then the documents seem to be particularly vague as to the TLB invalidation and what happens if the TLB is not properly invalidated.</p>
  
  <h2>Question</h2>
  
  <p>For the simplicity, one can assume that <strong>we are talking about a uniprocessor system</strong>. Also, it can be assumed that <strong>no task-switch is required after changing the page structures</strong>. (thus <code>INVLPG</code> is always supposedly at least as good choice as reloading the <code>CR3</code> register).</p>
  
  <p>The base assumption is that one would need to reload <code>CR3</code> after each change to page tables and page directories, and such a system would be correct. However, if one wants to avoid flushing the TLB needlessly, one needs answers to the 2 questions:</p>
  
  <ol>
  <li><p>Provided that <code>INVLPG</code> is supported on the ISA, after what kind of changes can one safely use it instead of reloading the <code>CR3</code>? E.g. "If one unmaps one page frame (set the corresponding table entry to not present), one can always use <code>INVLPG</code>"?</p></li>
  <li><p>What kind of changes one can do to the tables and directories without touching either <code>CR3</code> or executing <code>INVLPG</code>? E.g. "If a page is not mapped at all (not present), one can write a PTE with <code>Present=1</code> for it without flushing the TLB at all"?</p></li>
  </ol>
  
  <p>Even after reading a quite a load of ISA documents and everything related to <code>INVLPG</code> here on Stack Overflow I am not personally sure of either examples I presented there. Indeed, one <a href="https://stackoverflow.com/a/10671172/918959">notable post</a> stated it right away: "I don't know exactly when you should use it and when you shouldn't." Thus any certain, correct examples, preferably documented, and for either IA32 or x86-64, that you can give, are appreciated.</p>