title: How to understand byte pair encoding?
body: <p>I read a lot of tutorial about BPE but I am still confuse how it works.</p>
  
  <p>for example.
  In a tutorial online, they said the folowing : </p>
  
  <p>Algorithm</p>
  
  <p>Prepare a large enough training data (i.e. corpus)</p>
  
  <p>Define a desired subword vocabulary size</p>
  
  <p>Split word to sequence of characters and appending suffix “” to end of</p>
  
  <p>word with word frequency. So the basic unit is character in this stage. For example, the frequency of “low” is 5, then we rephrase it to “l o w ”: 5
      Generating a new subword according to the high frequency occurrence.
      Repeating step 4 until reaching subword vocabulary size which is defined in step 2 or the next highest frequency pair is 1.</p>
  
  <p>Taking “low: 5”, “lower: 2”, “newest: 6” and “widest: 3” as an example, the highest frequency subword pair is e and s. It is because we get 6 count from newest and 3 count from widest. Then new subword (es) is formed and it will become a candidate in next iteration.</p>
  
  <p>In the second iteration, the next high frequency subword pair is es (generated from previous iteration )and t. It is because we get 6count 
  from newest and 3 count from widest.</p>
  
  <p>I do not understand why low is 5   and lower is  2:</p>
  
  <p>does this meand  l , o, w , lo, ow +  = 6 and then lower  equal two but why is not e, r, er which gives three ?</p>