title: Python Scrapy Response replace unicodes
body: <p>I have a Scrapy request with the following response returned:</p>
  <pre><code>   {&quot;td_data&quot;:&quot;\n\n\t
  &lt;div class=\&quot;td-block-span12\&quot;&gt;\n\r\n
      &lt;div class=&quot;td_module_11 td_module_wrap td-animation-stack&quot; &gt;\r\n
      &lt;div class=\&quot;td-module-thumb\&quot;&gt;
          &lt;a href=\&quot;http:\/\/www.somesite.net\/2022\/02\/sometitle\/\&quot; rel=\&quot;bookmark\&quot; title=\&quot;&amp;#8222;sometitle?\&quot;&gt;
          &lt;\/a&gt;&lt;\/div&gt;\r\n
              &lt;div class=\&quot;item-details\&quot;&gt;\r\n
                  &lt;h3 class=\&quot;entry-title td-module-title\&quot;&gt;
                  &lt;\/h3&gt;\r\n
                  &lt;div class=\&quot;td-module-meta-info\&quot;&gt;\r\n
                      &lt;span class=\&quot;td-post-date\&quot;&gt;&lt;time class=\&quot;entry-date updated td-module-date\&quot; datetime=\&quot;2022-02-11T07:08:32+00:00\&quot; &gt;11. Februar 2022&lt;\/time&gt;&lt;\/span&gt;
                          &lt;div
              class=\&quot;td-module-comments\&quot;&gt;&lt;a
              href=\&quot;http:\/\/www.somesite.net\/2022\/02\/sometitle\/#comments\&quot;&gt;31&lt;\/a&gt;&lt;\/div&gt;                &lt;\/div&gt;\r\n\r\n                &lt;div
              class=\&quot;td-excerpt\&quot;&gt;\r\n                    \u201eSome excerpt text\u201c\u00a0\u2013                &lt;\/div&gt;\r\n\r\n                &lt;div
              class=\&quot;td-read-more\&quot;&gt;\r\n                    &lt;a
              href=\&quot;http:\/\/www.somesite.net\/2022\/02\/sometitle\/\&quot;&gt;Continue&lt;\/a&gt;\r\n                &lt;\/div&gt;\r\n            &lt;\/div&gt;\r\n\r\n        &lt;\/div&gt;\r\n\r\n        \n\t&lt;\/div&gt;
          &lt;!-- .\/td-block-span12 --&gt;\n\n\t
                              &lt;div class=\&quot;td-block-span12\&quot;&gt;\n\r\n
                                  &lt;div class=\&quot;td_module_11
                                                                                             td_module_wrap
                                                                                             td-animation-stack\&quot;&gt;\r\n            &lt;div
                  class=\&quot;td-module-thumb\&quot;&gt;&lt;a
                  href=\&quot;http:\/\/www.somesite.net\/2022\/02\/othertitle\/\&quot;
                  rel=\&quot;bookmark\&quot; title=\&quot;Some other title\&quot;&gt;&lt;\/div&gt;\r\n            &lt;div
                  class=\&quot;item-details\&quot;&gt;\r\n                &lt;h3 class=\&quot;entry-title td-module-title\&quot;&gt;&lt;\/h3&gt;\r\n                &lt;div
                  class=\&quot;td-module-meta-info\&quot;&gt;\r\n                                                            &lt;span
                  class=\&quot;td-post-date\&quot;&gt;&lt;time class=\&quot;entry-date updated td-module-date\&quot; datetime=\&quot;2022-02-10T20:12:20+00:00\&quot; &gt;10. Februar 2022&lt;\/time&gt;&lt;\/span&gt;                    &lt;div
                  class=\&quot;td-module-comments\&quot;&gt;&lt;a
                  href=\&quot;http:\/\/www.somesite.net\/2022\/02\/othertitle\/#comments\&quot;&gt;58&lt;\/a&gt;&lt;\/div&gt;                &lt;\/div&gt;\r\n\r\n                &lt;div
                  class=\&quot;td-excerpt\&quot;&gt;\r\n  Other excerpt                &lt;\/div&gt;\r\n\r\n                &lt;div
                  class=\&quot;td-read-more\&quot;&gt;\r\n                    &lt;a
                  href=\&quot;http:\/\/www.somesite.net\/2022\/02\/othertitle\/\&quot;&gt;continue&lt;\/a&gt;\r\n                &lt;\/div&gt;\r\n            &lt;\/div&gt;\r\n\r\n        &lt;\/div&gt;\r\n\r\n        \n\t&lt;\/div&gt;
              &lt;!-- .\/td-block-span12 --&gt;&quot;,&quot;td_block_id&quot;:&quot;td_uid_47_620641c2d6ce0&quot;,&quot;td_hide_prev&quot;:false,&quot;td_hide_next&quot;:false}
  </code></pre>
  <p>I don't know why but for some reason everything is marked with newline codings, unicodes etc. I tried to transform everything properly afterwards but I cannot replace the unicodes like \u201e and so one. I mean I could parse the string for every single of those (like looking specifically for \u201e and then the next one and so on), but this would be quite ugly and not robust compared to new codes. What I want to achieve is that the codes will be represented by their equivalent, so for example \u00f6 converts to ö.
  I also tried several encoding/decoding steps, but they didn't work. Here is my current scrapy callback method:</p>
  <pre><code>def pg_parse(self,response):
      print(response.text)
      response_formatted = (u'%s' % response.text)
      #response_formatted = response_formatted.encode(&quot;ascii&quot;, &quot;ignore&quot;).decode()
      response_formatted = response_formatted.encode(&quot;utf-8&quot;).decode()
      print(response_formatted)
      response_formatted = response_formatted.replace(&quot;\\r&quot;,&quot;&quot;).replace(&quot;\\n&quot;,&quot;&quot;).replace(&quot;\\t&quot;,&quot;&quot;).replace(&quot;                    &quot;,&quot;&quot;)
      #print(response_formatted)
      for sel in scrapy.Selector(text=response_formatted).xpath('//div[contains(@class, &quot;td_module_11&quot;)]'):
          title = sel.xpath('div[contains(@class,&quot;td-module-thumb&quot;)]/a/@title').extract()
          link = sel.xpath('div[contains(@class,&quot;td-module-thumb&quot;)]/a/@href').extract()
          time = sel.xpath('div[contains(@class,&quot;item-details&quot;)]/div[contains(@class,&quot;td-module-meta-info&quot;)]/span/time/@datetime').extract()
          comments = sel.xpath('div[contains(@class,&quot;item-details&quot;)]/div[contains(@class,&quot;td-module-meta-info&quot;)]/div/a/text()').extract()
          desc = [&quot;&quot;]# sel.extract()
          excerpt = sel.xpath('div[contains(@class,&quot;item-details&quot;)]/div[contains(@class,&quot;td-excerpt&quot;)]/text()').extract()
          print(title,link,time,comments,desc,excerpt)
          print(&quot;----------------&quot;)
  </code></pre>