title: scrapy with newline characters and nested tags
body: <p>Disclaimer: new to scrapy.</p>
  
  <p>I have a table with pretty irregular rows, The basic structure is:</p>
  
  <pre><code>&lt;tr&gt;
   &lt;td&gt; some text &lt;/td&gt;
   &lt;td&gt; some other text &lt;/td&gt;
   &lt;td&gt; yet some text &lt;/td&gt;
  &lt;/tr&gt;
  </code></pre>
  
  <p>but occasionally (a few hundred times) some rows are</p>
  
  <pre><code>&lt;tr&gt;
   &lt;td&gt; &lt;p&gt; some text &lt;p&gt; &lt;/td&gt;
   &lt;td&gt; &lt;div class="class-whateva"&gt; &lt;p&gt; some other text &lt;/p&gt;&lt;/div&gt; &lt;/td&gt;
   &lt;td&gt; &lt;span id="strange-id"&gt; 
    &lt;a href="somelink"&gt; yet some text &lt;/a&gt; 
      &lt;span&gt; &lt;/td&gt;
  &lt;/tr&gt;
  </code></pre>
  
  <p>or other permutations of 1 or 2 nested "p" "div" and "span" with or without return line characters. </p>
  
  <p>I've taken care of the nested "span span" or "p div" or "div span" with conditional statements of the form:</p>
  
  <pre><code>for row in allrows:
        if  row.select('td[2]/text()'):
              item['seconditem']=row.select('td[2]/text()').extract()
        elif row.select('td[2]/*/text()'):
              item['seconditem']=row.select('td[2]/*/text()').extract()
        elif row.select('td[2]/*/*/text()'):
              item['seconditem']=row.select('td[2]/*/*/text()').extract()
  </code></pre>
  
  <p>Now I have two questions:</p>
  
  <p>(1)  Is conditional</p>
  
  <pre><code>td[2]/*/*/text()
  </code></pre>
  
  <p>the right way to go about for irregular nested rows?</p>
  
  <p>(2) I am still missing all the cases where there is a return (or newline) before the tag.
  So if the row is of the form:</p>
  
  <pre><code>   &lt;td&gt;&lt;div&gt;
        &lt;p&gt;text &lt;/p&gt;
     &lt;/div&gt;&lt;/td&gt;
  </code></pre>
  
  <p>All my xpath will return is a ['\n     '].  Any trick to catch what's after the newline character?</p>
  
  <p>Any tips appreciated.  Thanks.</p>