{"action":"created","issue":{"url":"https://api.github.com/repos/learningequality/ka-lite/issues/3434","labels_url":"https://api.github.com/repos/learningequality/ka-lite/issues/3434/labels{/name}","comments_url":"https://api.github.com/repos/learningequality/ka-lite/issues/3434/comments","events_url":"https://api.github.com/repos/learningequality/ka-lite/issues/3434/events","html_url":"https://github.com/learningequality/ka-lite/issues/3434","id":65056142,"number":3434,"title":"Get rid of sys.setdefaultencoding and move to Unicode by default","user":{"login":"benjaoming","id":374612,"avatar_url":"https://avatars.githubusercontent.com/u/374612?v=3","gravatar_id":"","url":"https://api.github.com/users/benjaoming","html_url":"https://github.com/benjaoming","followers_url":"https://api.github.com/users/benjaoming/followers","following_url":"https://api.github.com/users/benjaoming/following{/other_user}","gists_url":"https://api.github.com/users/benjaoming/gists{/gist_id}","starred_url":"https://api.github.com/users/benjaoming/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benjaoming/subscriptions","organizations_url":"https://api.github.com/users/benjaoming/orgs","repos_url":"https://api.github.com/users/benjaoming/repos","events_url":"https://api.github.com/users/benjaoming/events{/privacy}","received_events_url":"https://api.github.com/users/benjaoming/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/learningequality/ka-lite/labels/new%20feature","name":"new feature","color":"bfdadc"},{"url":"https://api.github.com/repos/learningequality/ka-lite/labels/refactoring%20and%20performance","name":"refactoring and performance","color":"bfd4f2"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/learningequality/ka-lite/milestones/50","html_url":"https://github.com/learningequality/ka-lite/milestones/0.16.0","labels_url":"https://api.github.com/repos/learningequality/ka-lite/milestones/50/labels","id":1227915,"number":50,"title":"0.16.0","description":"","creator":{"login":"aronasorman","id":191955,"avatar_url":"https://avatars.githubusercontent.com/u/191955?v=3","gravatar_id":"","url":"https://api.github.com/users/aronasorman","html_url":"https://github.com/aronasorman","followers_url":"https://api.github.com/users/aronasorman/followers","following_url":"https://api.github.com/users/aronasorman/following{/other_user}","gists_url":"https://api.github.com/users/aronasorman/gists{/gist_id}","starred_url":"https://api.github.com/users/aronasorman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aronasorman/subscriptions","organizations_url":"https://api.github.com/users/aronasorman/orgs","repos_url":"https://api.github.com/users/aronasorman/repos","events_url":"https://api.github.com/users/aronasorman/events{/privacy}","received_events_url":"https://api.github.com/users/aronasorman/received_events","type":"User","site_admin":false},"open_issues":125,"closed_issues":134,"state":"open","created_at":"2015-07-27T17:17:39Z","updated_at":"2016-02-01T19:41:21Z","due_on":"2015-12-30T08:00:00Z","closed_at":null},"comments":7,"created_at":"2015-03-29T12:37:07Z","updated_at":"2016-02-01T19:42:16Z","closed_at":null,"body":"In `kalite.settings` there's the following block of code (some modifications because of the setuptools installer)\r\n\r\n```python\r\n# set the default encoding\r\n# OK, so why do we reload sys? Because apparently sys.setdefaultencoding\r\n# is deleted somewhere at startup. Reloading brings it back.\r\n# see: http://blog.ianbicking.org/illusive-setdefaultencoding.html\r\n\r\n# benjaoming: We need to somehow get rid of this as it's discouraged\r\n# http://stackoverflow.com/questions/3828723/why-we-need-sys-setdefaultencodingutf-8-in-a-py-script\r\n# http://ziade.org/2008/01/08/syssetdefaultencoding-is-evil/\r\ntry:\r\n    DEFAULT_ENCODING = DEFAULT_ENCODING\r\nexcept NameError:\r\n    from django.conf.settings import DEFAULT_ENCODING\r\n\r\nif sys.getdefaultencoding() != DEFAULT_ENCODING:\r\n    reload(sys)\r\n    sys.setdefaultencoding(DEFAULT_ENCODING)\r\n```\r\n\r\nThe motivation behind this is memory optimization. Experience reported by @aronasorman and @66eli77 says that memory efficiency is ~1.5 and this is much crucial for Raspberry Pi execution.\r\n\r\nIndeed, this is also true... but it's very sensitive to the string size. This is pretty crazy, because the [Python 2 docs](https://docs.python.org/2/howto/unicode.html#encodings) say the following...\r\n\r\n> UTF-8 is fairly compact; the majority of code points are turned into two bytes, and values less than 128 occupy only a single byte.\r\n\r\nLet's do 3 characters of ascii....\r\n\r\n```\r\n>>> from sys import getsizeof\r\n>>> getsizeof(\"123\")\r\n40\r\n>>> getsizeof(u\"123\")\r\n64\r\n```\r\n\r\nBut then, lets do a lot more...\r\n\r\n```\r\n>>> getsizeof(\"123\" * 1000)\r\n3037\r\n>>> getsizeof(u\"123\" * 1000)\r\n12052\r\n```\r\n\r\nNow the unicode type is consuming 4x as much space!? Let's do something with unicode characters...\r\n\r\n```\r\n>>> getsizeof(\"®€łœßĸ“đłłđ®\" * 1000)\r\n26037\r\n>>> getsizeof(u\"®€łœßĸ“đłłđ®\" * 1000)\r\n48052\r\n```\r\n\r\nNow it's less than 2x as much...\r\n\r\nThe problem will go away in Python 3.. as I understand it, the unicode (which is the way that strings will be treated by default), is also optimized for choosing a better codepoint something something...\r\n\r\n```\r\nPython 3.4.0 (default, Apr 11 2014, 13:05:11) \r\n[GCC 4.8.2] on linux\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> from sys import getsizeof\r\n>>> getsizeof(\"123\")\r\n52\r\n>>> getsizeof(\"123\" * 1000)\r\n3049\r\n```\r\n\r\nAs you see, Python 3, perfectly handles UTF-8 encoding.\r\n\r\nNow, here's the bible of all this unicode stuff:\r\n\r\nhttps://docs.python.org/2/howto/unicode.html\r\n\r\n# What to do!!?\r\n\r\nI suggest moving everything to Py2+3 compatibility, which will mean using `from __future__ import unicode_literals`. From my testing, this does __not__ solve the memory issues, but it's a necessary step to start including Python 3 compatibility.\r\n\r\nThis would leave us with a memory problem, but we have to solve it by reducing our dependency on loading big JSON files in memory. This is what databases are used for :D\r\n\r\nAnother issue solved by forcing the default encoding to `UTF-8` has been to avoid explicitly encoding/decoding `str` objects. This should also be solved by moving to `unicode_literals` on Python 2."},"comment":{"url":"https://api.github.com/repos/learningequality/ka-lite/issues/comments/178149360","html_url":"https://github.com/learningequality/ka-lite/issues/3434#issuecomment-178149360","issue_url":"https://api.github.com/repos/learningequality/ka-lite/issues/3434","id":178149360,"user":{"login":"rtibbles","id":1680573,"avatar_url":"https://avatars.githubusercontent.com/u/1680573?v=3","gravatar_id":"","url":"https://api.github.com/users/rtibbles","html_url":"https://github.com/rtibbles","followers_url":"https://api.github.com/users/rtibbles/followers","following_url":"https://api.github.com/users/rtibbles/following{/other_user}","gists_url":"https://api.github.com/users/rtibbles/gists{/gist_id}","starred_url":"https://api.github.com/users/rtibbles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rtibbles/subscriptions","organizations_url":"https://api.github.com/users/rtibbles/orgs","repos_url":"https://api.github.com/users/rtibbles/repos","events_url":"https://api.github.com/users/rtibbles/events{/privacy}","received_events_url":"https://api.github.com/users/rtibbles/received_events","type":"User","site_admin":false},"created_at":"2016-02-01T19:42:16Z","updated_at":"2016-02-01T19:42:16Z","body":"Given that we are moving these strings out of memory now, hopefully this can be addressed in 0.16 still."}}