{"action":"opened","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/3272","repository_url":"https://api.github.com/repos/scrapy/scrapy","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/3272/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/3272/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/3272/events","html_url":"https://github.com/scrapy/scrapy/issues/3272","id":325740843,"number":3272,"title":"KeyError with the initialization of an Item Field defined with None using ItemLoader","user":{"login":"NadirRisc","id":38318897,"avatar_url":"https://avatars3.githubusercontent.com/u/38318897?v=4","gravatar_id":"","url":"https://api.github.com/users/NadirRisc","html_url":"https://github.com/NadirRisc","followers_url":"https://api.github.com/users/NadirRisc/followers","following_url":"https://api.github.com/users/NadirRisc/following{/other_user}","gists_url":"https://api.github.com/users/NadirRisc/gists{/gist_id}","starred_url":"https://api.github.com/users/NadirRisc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NadirRisc/subscriptions","organizations_url":"https://api.github.com/users/NadirRisc/orgs","repos_url":"https://api.github.com/users/NadirRisc/repos","events_url":"https://api.github.com/users/NadirRisc/events{/privacy}","received_events_url":"https://api.github.com/users/NadirRisc/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-23T14:47:17Z","updated_at":"2018-05-23T14:47:17Z","closed_at":null,"author_association":"NONE","body":"Using Scrapy 1.5.0\r\nI took a look at the FAQ section and nothing was relevant about it.\r\nSame for issues with keyword `KeyError` on github, Reddit, or GoogleGroups.\r\n\r\nAs you can see below, it seems to me that here is an inconsistency when we load an `Item` or initialize it with a values as `None` or an empty string. First we add a value to our field (here title) through a `ItemLoader`. Then the loader creates an item with the `load_item()` method. Once it's done we can't access the field if the value was `None` or an empty string. The inconsistency is that the other method (initializing an `Item` directly with a field set to `None` or empty string) doesn't raise a `KeyError` (field is set).\r\n\r\nThe class `TakeFirst` [don't return any value when they're set with `None` or an empty string](/scrapy/scrapy/tree/master/scrapy/loader/processors.py#L57). Which prevents the method `load_item()` in `ItemLoader` class [to add an entry to the field.](/scrapy/scrapy/tree/master/scrapy/loader/__init__.py#L116)\r\n\r\n\r\nHere is a minimal source code that represents the inconsistency.\r\n\r\n```python\r\nimport scrapy\r\nfrom scrapy.loader import ItemLoader\r\nfrom scrapy.loader.processors import TakeFirst, MapCompose\r\n\r\nclass MyItem(scrapy.Item):\r\n    title = scrapy.Field()\r\n\r\nclass MyItemLoader(ItemLoader):\r\n    default_output_processor = TakeFirst()\r\n    title_in = MapCompose()\r\n\r\nclass MySpider(scrapy.Spider):\r\n    name = \"My\"\r\n    start_urls = ['https://blog.scrapinghub.com']\r\n\r\n    def parse(self, response):\r\n        titles = ['fake title', '', None]\r\n\r\n        for title in titles:\r\n            # First case: with ItemLoader\r\n            loader = MyItemLoader(item=MyItem())\r\n            loader.add_value('title', title)\r\n            loaded_item = loader.load_item()\r\n\r\n            # Second case: without ItemLoader\r\n            item = MyItem(title=title)\r\n\r\n            if title in ('', None):\r\n                # inconsistency!\r\n                assert not 'title' in loaded_item\r\n                assert 'title' in item\r\n```\r\n\r\nWe're using `Python 3.5` for our project and found the following workaround to prevent this error. \r\nWe introduce a new class (`DefaultAwareItem`) which fulfills unset fields were `default metadata` has been set previously.\r\n\r\n```python\r\nimport scrapy\r\n\r\nclass DefaultAwareItem(scrapy.Item):\r\n    \"\"\"Item class aware of 'default' metadata of its fields.\r\n\r\n    For instance to work, each field, which must have a default value, must\r\n    have a new `default` parameter set in field constructor, e.g.::\r\n\r\n        class MyItem(DefaultAwareItem):\r\n            my_defaulted_field = scrapy.Field()\r\n            # Identical to:\r\n            #my_defaulted_field = scrapy.Field(default=None)\r\n            my_other_defaulted_field = scrapy.Field(default='a value')\r\n\r\n    \"\"\"\r\n    def __init__(self, *args, **kwargs):\r\n        super().__init__(*args, **kwargs)\r\n\r\n        for field_name, field_metadata in self.fields.items():\r\n            self.setdefault(field_name, field_metadata.get('default'))\r\n\r\nclass MyItem(DefaultAwareItem):\r\n    title = scrapy.Field()\r\n    title_explicitely_set = scrapy.Field(default=\"empty title\")\r\n```"}}
{"actor":{"display_login":"NadirRisc"}}