title: Collecting different attributes when crawling a website
body: <p>I've been creating spiders for different websites containing articles that output the raw text, text and URLs. I would like to get some more information, such as description, language, publishing date etc.</p>
  
  <p>This is one example:</p>
  
  <pre><code>import scrapy
  import re
  from nltk.corpus import stopwords
  from nltk.tokenize import word_tokenize
  import requests
  from bs4 import BeautifulSoup
  stop_words = set(stopwords.words("german"))
  
  
  class AListSpider(scrapy.Spider):
      name = 'aList'
      r = requests.get("https://www.a-list.at/newssitemap.xml")
      xml = r.text
      soup = BeautifulSoup(xml)
      urls_wien = list(filter(lambda y: '/wien/' in y, map(lambda x: x.text, soup.find_all("loc"))))
  
      def start_requests(self):
          urls = self.urls_wien
          for url in urls:
              yield scrapy.Request(url=url, callback=self.parse)
  
      def parse(self, response):
          try:
              yield {
                'raw_text': BeautifulSoup(response.css('.news-single-item').get(), "html.parser").find(class_="news-single-item").text,
                'text': ' '.join([i for i in word_tokenize(re.sub(pattern='[^a-zA-Z_\-ÖöÜüÄäßèé]',string=BeautifulSoup(response.css('.news-single-item').get(), "html.parser").find(class_="news-single-item").text, repl=' ')) if i not in stop_words and not re.match('[0-9]', i) and len(i) &gt;1]),
                'url': response.url
            }
          except:
              pass
  </code></pre>
  
  <p>I have read that it is possible to collect more information using schema.org, could anyone please explain how this works? </p>