title: Can we use Xidel for extracting data across a site into search files?
body: <p>Background: We are aggregating content from some websites (with permission) for use in supplementary search functions for another application. An example is the news section of <a href="https://centenary.bahai.us" rel="nofollow noreferrer">https://centenary.bahai.us</a>. We thought to use xidel for this purpose, since the template file paradigm seems an elegant way to extract data from html, e.g. for a template:</p>
  
  <pre><code>&lt;h1 class="title"&gt;{$title}&lt;/h1&gt;?
  &lt;div class="node build-mode-full"&gt;
    {$url:=$url}
    &lt;div class="field-field-audio"&gt;?
      &lt;audio src="{$audio:='https://' || $host || .}"&gt;&lt;/audio&gt;?
    &lt;/div&gt;?
    &lt;div class="field-field-clip-img"&gt;
      &lt;a href="{$image:='https://' || $host || .}" class="imagefield-field_clip_img"&gt;&lt;/a&gt;*
    &lt;/div&gt;?
    &lt;div class="field-field-pubname"&gt;{$publication}&lt;/div&gt;?
    &lt;div class="field-field-historical-date"&gt;{$date}&lt;/div&gt;?
    &lt;div class="location"&gt;&lt;div class="adr"&gt;{$location}&lt;/div&gt;?&lt;/div&gt;?
    &lt;div class="node-body"&gt;{$text}&lt;/div&gt;
  &lt;/div&gt;?
  </code></pre>
  
  <p>...we can run a command like the following:</p>
  
  <pre><code>xidel "https://centenary.bahai.us" -e "$(&lt; template.html)" -f "//a[contains(@href, '/news/')]" --silent --color=never --output-format=json-wrapped &gt; index.json
  </code></pre>
  
  <p>...which will give us json formatted data from all the news pages on centenary.bahai.us. An example article would look like this:</p>
  
  <pre><code>{
  "title": "Bahá’ísm the Religion of Brotherhood", 
  "url": "https://centenary.bahai.us/news/bahaism-religion-brotherhood", 
  "audio": "https://centenary.bahai.us/sites/default/files/453_0.mp3", 
  "image": "https://centenary.bahai.us/sites/default/files/imagecache/lightbox-large/images/press_clippings/03-31-1912_NYT_Bahaism_the_Religion_of_Brotherhood.png", 
  "publication": "The New York Times", 
  "date": "March 31, 1912", 
  "location": "New York, NY", 
  "text": "A posthumous volume of “Essays in Radical Empiricism,” by William James, will be published in April by Longmans, Green &amp; Co. This house will also bring out “Leo XIII, and Anglican Orders,” by Viscount Halifax, and “Bahá’ísm, the Religion of Brotherhood, and Its Place in the Evolution of Creeds,” by Francis H. Skrine. In the latter an analysis is made of the Gospel of Bahá’u’lláh and his successor. ‘Abdu’l-Bahá — whose arrival in this country is expected early in April — and a forecast is attempted of its influence on civilization."
  },
  </code></pre>
  
  <p>That's just beautiful, and a whole lot easier than some mishmash of httrack and pup or (God forbid) sed and regex, but there are a few issues:</p>
  
  <ol>
  <li>We would like to have separate files for each document, whereas this gives us one large json file.</li>
  <li>Even with the <code>--silent</code> flag, we still get status messages in the output which invalidate the json, such as <code>**** Retrieving (GET): https://centenary.bahai.us ****</code> or <code>**** Processing: https://centenary.bahai.us/ ****</code> or <code>** Current variable state: **</code></li>
  <li>The process seems too fragile; if there is any discrepancy between the template and the actual html, the entire process errors out and we get nothing. We would prefer to have it output an error for only the one page and then continue with the next URL.</li>
  </ol>
  
  <p>Xidel seems like a game-changing tool which should make this job possible with a one-line command and a simple extract template file; what am I missing here?</p>