title: How to parse large amount of data passed to kernel module through /proc file?
body: <p><strong>Edit:</strong> I have found <code>seq_file</code> that eases writing a lot of data from kernel to user-space. What I am looking for is the opposite; an API that facilitates reading a lot of data (more than one page) from user-space.</p>
  
  <p><strong>Edit 2</strong>: I am implementing a port of <code>&lt;stdio.h&gt;</code> as a kernel module that would be able to open <code>/proc</code> (and later, other virtual file systems) similar to <code>FILE</code>s and handle input and output similar to <code>&lt;stdio.h&gt;</code>. You can find the project <a href="https://github.com/ShabbyX/kio" rel="nofollow noreferrer">here</a>.</p>
  
  <hr>
  
  <p>I have found a LOT of questions on how the kernel can write large amounts of data to /proc (for user-space programs to take), but nothing for the other way around. Let me elaborate:</p>
  
  <p>This question is basically about the algorithm by which the input is tokenized (for example to <code>int</code>s or a mixture of <code>int</code> and string etc), <strong>given that the data maybe broken between multiple buffers</strong>.</p>
  
  <p>For example, imagine the following data is being sent to the kernel module:</p>
  
  <pre><code>12345678 81234567 78123456 67812345 5678 1234 45678123 3456 7812 23456781
  </code></pre>
  
  <p>and for the sake of this example, let's say the page size by which Linux feeds the /proc handler is 20 bytes (vs the real 4KB).</p>
  
  <p>The function that reads the data from /proc (in the kernel module) then sees the data as such:</p>
  
  <pre><code>call 1:
  "12345678 81234567 78"
  call 2:
  "123456 67812345 5678"
  call 3:
  " 1234 45678123 3456 "
  call 4:
  "7812 23456781"
  </code></pre>
  
  <p>As you can see, when <code>78</code> is read in the first call, it shouldn't be processed yet until the next frames for it to decide whether <code>78</code> was a whole number or one cut between frames.</p>
  
  <p>Now I found <a href="http://www.xenotime.net/linux/doc/seq_file_howto.txt" rel="nofollow noreferrer"><code>seq_file</code></a>s that apparently are only for when the kernel wants to <em>write</em> data to user rather than <em>read</em> (or it could be that the HOWTO is horribly written).</p>
  
  <h2>What I have done</h2>
  
  <p>So far, I have come with the following solution (I am writing from memory, so I may miss a couple error checkings, but bear with me):</p>
  
  <p>In the initialization phase (say <code>init_module</code>):</p>
  
  <pre><code>initialize mutex1 to 1 and mutex2 to 0
  create /proc entry
  call data_processor
  </code></pre>
  
  <p>/proc reader:</p>
  
  <pre><code>1. down(mutex1)    /* down_interruptible of course, but let's not get into details */
  
  2. copy_from_user to an internal buffer
     buffer_index = 0
     data_length = whatever the size is
  
  3. strip spaces from end of buffer (except if all left from buffer is 1 space)
     if so, there_was_space_after = 1 else 0
  
  4. up(mutex2)
  </code></pre>
  
  <p>I will explain why I strip spaces later</p>
  
  <p><code>get_int</code> function:</p>
  
  <pre><code>wait_for_next = 0
  number_was_cut = 0
  last_number = 0
  
  do
  {
      1. down(mutex2)
  
      2. if (number_was_cut &amp;&amp; !isdigit(buffer[buffer_index]))
             break     /* turns out it wasn't really cut
                          as beginning of next buffer is ' ' */
         number_was_cut = 0
         wait_for_next = 0
  
      3. while (buffer_index &lt; data_length &amp;&amp; !isdigit(buffer_index[buffer_index]))
             ++buffer_index;    /* skip white space */
  
      4. while (buffer_index &lt; data_length &amp;&amp; isdigit(buffer[buffer_index]))
             last_number = last_number * 10 + buffer[buffer_index++] - '0';
  
      5. if (buffer_index &gt;= data_length &amp;&amp; !there_was_space_after)
             number_was_cut = 1
             wait_for_next = 1
             up(mutex1)         /* let more data come in */
         else
             up(mutex2)         /* let get_int continue */
             break
  } while (wait_for_next)
  
  return last_number
  </code></pre>
  
  <p><code>data_processor</code> function (for example):</p>
  
  <pre><code>int first_num = get_int()
  int sencod_num = get_int()
  for i = first_num to second_num
      do_whatever(get_int())
  </code></pre>
  
  <p><strong>Explanation:</strong> First, see <code>data_processor</code>. It doesn't get involved in complications on how the data are read, so it just gets integers and does whatever it wants with them. Now let's see /proc reader. It basically waits for <code>data_processor</code> to call <code>get_int</code> enough times for all current data to be consumed (step 1) and then copies the next buffer into internal memory, allowing <code>data_processor</code> to continue (step 2). It then needs to strip trailing spaces so <code>get_int</code> could be simplified a bit (step 3). Finally, it signals <code>get_int</code> that it can start reading the data (step 4).</p>
  
  <p>The <code>get_int</code> function first waits for data to arrive (step 1), (ignore step 2 for now) it skips any unwanted characters (step 3) and then starts reading the number (step 4). The end of reading the number is by two possibilities; the end of buffer is reached (in which case, if /proc reader had not stripped any spaces, then the number <strong>could</strong> be cut between frames) or white space is met. In the former case, it needs to signal /proc reader to read in more data and wait for another cycle to append the rest of the number to the current one and in the later case, it returns the number (step 5). If continuing from last frame, check to see if new frame starts with a number or not. If not, then previous number was actually a whole number and should be returned. Otherwise, it needs to continue appending digits to last number (step 2).</p>
  
  <h2>Problem</h2>
  
  <p><strong>The main problem with this method is that it is overly complicated</strong>. It gets much more complicated when <code>get_string</code> is added, or the read integer could be hex etc. Basically, you have to reinvent <code>sscanf</code>! Note that, <code>sscanf</code> could be used in this simple example at step 4 of <code>get_int</code> instead of the <code>while</code> loop (or also with <code>get_string</code>, but that gets more tricky when hex input is also possible (imagine the hex number being cut between 0 and x0212ae4). Even so, it just replaces step 4 of <code>get_int</code> and the rest of the stuff should still remain.</p>
  
  <p>It actually got me many bugs and heavy testing to perfect all the special cases. That's another reason why it doesn't look elegant to me.</p>
  
  <h2>Questions</h2>
  
  <p>I would like to know if there is any better method to handle this. I am aware that using shared memory could be an option, but I'm looking for an algorithm for this task (more out of curiosity since I already have my working solution). More specifically:</p>
  
  <ul>
  <li>Is there an already implemented method in the Linux kernel that can be treated like a normal C <code>FILE</code> from which you can take data and it handles the breaking of data into pages itself?</li>
  <li>If no, am I over-complicating things and am I missing an obvious simple solution?</li>
  <li>I believe <code>fscanf</code> faces a similar problem. How is this handled by that?</li>
  </ul>
  
  <p>Side question: Is it a terrible thing that I'm blocking the /proc reader on a mutex? I mean, writing data can be blocking, but I'm not sure if that normally happens in user-space or kernel-space.</p>