<h3>What happens?</h3>
<p>As you already mentioned - Seems to bee not well formed in soup cause it is missing the <code>&lt;link&gt;</code> it has only the <code>&lt;/link&gt;</code>, so you wont get text out of it with <code>text</code> attribute.</p>
<p>But good news there is a solution.</p>
<h3>How to fix?</h3>
<p>Just select the text with <code>next_sibling</code> of the <code>&lt;link&gt;</code> element:</p>
<pre><code>i.find('link').next_sibling
</code></pre>
<h3>Output</h3>
<pre><code>[{&quot;title&quot;: &quot;Gitlab from YC to IPO&quot;, &quot;link&quot;: &quot;https://blog.ycombinator.com/gitlab-from-yc-to-ipo/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 13:31:43 +0000&quot;}, {&quot;title&quot;: &quot;Apple Joins Blender Development Fund&quot;, &quot;link&quot;: &quot;https://www.blender.org/press/apple-joins-blender-development-fund/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 14:48:59 +0000&quot;}, {&quot;title&quot;: &quot;Sunset Geometry (2016)&quot;, &quot;link&quot;: &quot;https://www.shapeoperator.com/2016/12/12/sunset-geometry/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 14:29:08 +0000&quot;}, {&quot;title&quot;: &quot;iPhone Macro: A Big Day for Small Things&quot;, &quot;link&quot;: &quot;https://lux.camera/iphone-macro-camera-a-big-day-for-small-things/&quot;, &quot;date&quot;: &quot;Mon, 11 Oct 2021 10:22:06 +0000&quot;}, {&quot;title&quot;: &quot;Michelin Airless&quot;, &quot;link&quot;: &quot;https://www.michelin.com/en/innovation/vision-concept/airless/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 14:36:58 +0000&quot;}, {&quot;title&quot;: &quot;Release (YC W20) Is Hiring \u2013 Product Marketing Manager&quot;, &quot;link&quot;: &quot;https://releasehub.com/company#hire&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 17:00:15 +0000&quot;}, {&quot;title&quot;: &quot;Global Climate Report \u2013 September 2021&quot;, &quot;link&quot;: &quot;https://www.ncdc.noaa.gov/sotc/global/202109&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 14:49:59 +0000&quot;}, {&quot;title&quot;: &quot;Esbuild \u2013 An extremely fast JavaScript bundler&quot;, &quot;link&quot;: &quot;https://esbuild.github.io/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 05:07:27 +0000&quot;}, {&quot;title&quot;: &quot;Small Language Models Are Also Few-Shot Learners&quot;, &quot;link&quot;: &quot;https://aclanthology.org/2021.naacl-main.185/&quot;, &quot;date&quot;: &quot;Tue, 12 Oct 2021 09:59:34 +0000&quot;}, {&quot;title&quot;: &quot;Who was Aleph Null? (2013)&quot;, &quot;link&quot;: &quot;http://bit-player.org/2013/who-was-aleph-null&quot;, &quot;date&quot;: &quot;Mon, 11 Oct 2021 08:35:29 +0000&quot;}, {&quot;title&quot;: &quot;Hands-On Rust: Effective Learning Through 2D Game Development and Play&quot;, &quot;link&quot;: &quot;https://pragprog.com/titles/hwrust/hands-on-rust/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 07:59:24 +0000&quot;}, {&quot;title&quot;: &quot;Ask HN: What's the Point of Life?&quot;, &quot;link&quot;: &quot;https://news.ycombinator.com/item?id=28866558&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 16:38:15 +0000&quot;}, {&quot;title&quot;: &quot;What I wish I knew when learning F#&quot;, &quot;link&quot;: &quot;https://danielbachler.de/2020/12/23/what-i-wish-i-knew-when-learning-fsharp.html&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 12:07:40 +0000&quot;}, {&quot;title&quot;: &quot;Investing in Startups by Passing the Series 65&quot;, &quot;link&quot;: &quot;https://www.natecation.com/accredited-investor-investing-startups-series-65/&quot;, &quot;date&quot;: &quot;Wed, 13 Oct 2021 17:57:25 +0000&quot;}, {&quot;title&quot;: &quot;OpenBSD 7.0&quot;, &quot;link&quot;: &quot;https://www.openbsd.org/70.html&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 10:24:21 +0000&quot;}, {&quot;title&quot;: &quot;Countries are gathering in an effort to stop a biodiversity collapse&quot;, &quot;link&quot;: &quot;https://www.nytimes.com/2021/10/14/climate/un-biodiversity-conference-climate-change.html&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 13:32:00 +0000&quot;}, {&quot;title&quot;: &quot;Alden Global Capital, the secretive hedge fund gutting newsrooms&quot;, &quot;link&quot;: &quot;https://www.theatlantic.com/magazine/archive/2021/11/alden-global-capital-killing-americas-newspapers/620171/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 15:17:06 +0000&quot;}, {&quot;title&quot;: &quot;Child suicides in Japan hit record high&quot;, &quot;link&quot;: &quot;https://www3.nhk.or.jp/nhkworld/en/news/20211013_19/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 08:52:39 +0000&quot;}, {&quot;title&quot;: &quot;Every search bar looks like a URL bar to users&quot;, &quot;link&quot;: &quot;https://shkspr.mobi/blog/2021/10/every-search-bar-looks-like-a-url-bar-to-users/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 13:27:58 +0000&quot;}, {&quot;title&quot;: &quot;Psychonetics: A nerd's toolset to work with mind and perception&quot;, &quot;link&quot;: &quot;http://deconcentration-of-attention.com/psychonetics.html&quot;, &quot;date&quot;: &quot;Tue, 12 Oct 2021 11:28:43 +0000&quot;}, {&quot;title&quot;: &quot;FB seals off some internal message boards to prevent leaking, immediately leaked&quot;, &quot;link&quot;: &quot;https://www.businessinsider.com/facebook-whistleblower-leaks-restricts-staff-access-message-boards-elections-safety-2021-10&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 11:09:08 +0000&quot;}, {&quot;title&quot;: &quot;Working around expired root certificates&quot;, &quot;link&quot;: &quot;https://scotthelme.co.uk/should-clients-care-about-the-expiration-of-a-root-certificate/&quot;, &quot;date&quot;: &quot;Mon, 11 Oct 2021 21:27:27 +0000&quot;}, {&quot;title&quot;: &quot;An unprecedented wave of online bank fraud is hitting Britain&quot;, &quot;link&quot;: &quot;https://www.reuters.com/world/uk/welcome-britain-bank-scam-capital-world-2021-10-14/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 09:57:39 +0000&quot;}, {&quot;title&quot;: &quot;Interoperable Serendipity&quot;, &quot;link&quot;: &quot;https://noeldemartin.com/blog/interoperable-serendipity&quot;, &quot;date&quot;: &quot;Wed, 13 Oct 2021 12:02:37 +0000&quot;}, {&quot;title&quot;: &quot;Instagram took down post with figure from paper showing male advantage in sports&quot;, &quot;link&quot;: &quot;https://twitter.com/SwipeWright/status/1448064426670583814&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 16:36:30 +0000&quot;}, {&quot;title&quot;: &quot;IoT hacking and rickrolling my high school district&quot;, &quot;link&quot;: &quot;https://whitehoodhacker.net/posts/2021-10-04-the-big-rick&quot;, &quot;date&quot;: &quot;Tue, 12 Oct 2021 19:38:06 +0000&quot;}, {&quot;title&quot;: &quot;Boeing says certain 787 parts improperly manufactured&quot;, &quot;link&quot;: &quot;https://www.reuters.com/business/aerospace-defense/boeing-deals-with-new-defect-787-dreamliner-wsj-2021-10-14/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 13:26:46 +0000&quot;}, {&quot;title&quot;: &quot;Practice Problems for Hardware Engineers&quot;, &quot;link&quot;: &quot;https://arxiv.org/abs/2110.06526&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 03:48:24 +0000&quot;}, {&quot;title&quot;: &quot;Interface ergonomics: automation isn't just about time saved&quot;, &quot;link&quot;: &quot;https://macoy.me/blog/programming/InterfaceFriction&quot;, &quot;date&quot;: &quot;Wed, 13 Oct 2021 01:05:52 +0000&quot;}, {&quot;title&quot;: &quot;Syncthing \u2013 a continuous file synchronization program&quot;, &quot;link&quot;: &quot;https://syncthing.net/&quot;, &quot;date&quot;: &quot;Thu, 14 Oct 2021 01:23:19 +0000&quot;}]
</code></pre>
Different output from srcaping rss by using requests get and beautifulsoup
<p>I wanna scrape the data from code of this link: <a href="https://news.ycombinator.com/rss" rel="nofollow noreferrer">https://news.ycombinator.com/rss</a>. It includes the html syntax: &quot;link&gt;the URL&lt;/link' (It's full of open and close &lt;&gt; but cannot put it in here).
However, when using this code, the printed output of the link is: 'link/&gt;the URL' and there are no content of the key 'link' in json file.</p>
<pre><code>import requests
import bs4
from bs4 import BeautifulSoup
import json
import html5lib 

def rss(x):
r = requests.get(x)
s = BeautifulSoup(r.content, features='html5lib')
the_list = []
for i in s.find_all('item'):
    title = i.find('title').text
    link = i.find('link').text
    date = i.find('pubdate').text

    article = {
        'title' : title,
        'link' : link,
        'date' : date
    }

    the_list.append(article)
with open('the_list.json','w') as f:
    json.dump(the_list,f)

rss('https://news.ycombinator.com/rss')
</code></pre>