{"action":"created","issue":{"url":"https://api.github.com/repos/nengo/nengo_extras/issues/34","repository_url":"https://api.github.com/repos/nengo/nengo_extras","labels_url":"https://api.github.com/repos/nengo/nengo_extras/issues/34/labels{/name}","comments_url":"https://api.github.com/repos/nengo/nengo_extras/issues/34/comments","events_url":"https://api.github.com/repos/nengo/nengo_extras/issues/34/events","html_url":"https://github.com/nengo/nengo_extras/issues/34","id":191853451,"number":34,"title":"Improved accuracy in higher dimensions with alternate intercept distribution","user":{"login":"tcstewar","id":2380772,"avatar_url":"https://avatars.githubusercontent.com/u/2380772?v=3","gravatar_id":"","url":"https://api.github.com/users/tcstewar","html_url":"https://github.com/tcstewar","followers_url":"https://api.github.com/users/tcstewar/followers","following_url":"https://api.github.com/users/tcstewar/following{/other_user}","gists_url":"https://api.github.com/users/tcstewar/gists{/gist_id}","starred_url":"https://api.github.com/users/tcstewar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tcstewar/subscriptions","organizations_url":"https://api.github.com/users/tcstewar/orgs","repos_url":"https://api.github.com/users/tcstewar/repos","events_url":"https://api.github.com/users/tcstewar/events{/privacy}","received_events_url":"https://api.github.com/users/tcstewar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"jgosmann","id":850157,"avatar_url":"https://avatars.githubusercontent.com/u/850157?v=3","gravatar_id":"","url":"https://api.github.com/users/jgosmann","html_url":"https://github.com/jgosmann","followers_url":"https://api.github.com/users/jgosmann/followers","following_url":"https://api.github.com/users/jgosmann/following{/other_user}","gists_url":"https://api.github.com/users/jgosmann/gists{/gist_id}","starred_url":"https://api.github.com/users/jgosmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jgosmann/subscriptions","organizations_url":"https://api.github.com/users/jgosmann/orgs","repos_url":"https://api.github.com/users/jgosmann/repos","events_url":"https://api.github.com/users/jgosmann/events{/privacy}","received_events_url":"https://api.github.com/users/jgosmann/received_events","type":"User","site_admin":false},"assignees":[{"login":"jgosmann","id":850157,"avatar_url":"https://avatars.githubusercontent.com/u/850157?v=3","gravatar_id":"","url":"https://api.github.com/users/jgosmann","html_url":"https://github.com/jgosmann","followers_url":"https://api.github.com/users/jgosmann/followers","following_url":"https://api.github.com/users/jgosmann/following{/other_user}","gists_url":"https://api.github.com/users/jgosmann/gists{/gist_id}","starred_url":"https://api.github.com/users/jgosmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jgosmann/subscriptions","organizations_url":"https://api.github.com/users/jgosmann/orgs","repos_url":"https://api.github.com/users/jgosmann/repos","events_url":"https://api.github.com/users/jgosmann/events{/privacy}","received_events_url":"https://api.github.com/users/jgosmann/received_events","type":"User","site_admin":false}],"milestone":null,"comments":33,"created_at":"2016-11-27T10:13:56Z","updated_at":"2016-12-05T22:11:45Z","closed_at":null,"body":"So, Aaron's talk reminded me about something I'd wanted to look at ages ago: in high dimensions, it might not make sense to have a uniform distribution of intercepts.  A neuron with an intercept of 0.8 in 16-dimensions will only fire for a very small proportion of points in the 16-D space (0.00196%, as it turns out). And this also means a neuron with an intercept of -0.8 will fire for 99.998% of the space.  This means that those 40% of neurons (and likely many more) are kind of useless.\r\n\r\nSo, I finally sat down an analyzed this and figured out the math of what the intercept distribution should be such that the proportion of the space for which a neuron is active is a uniform distribution.  It turns out the transformation is\r\n\r\n```sqrt(1-betaincinv((d+1)/2, 1/2, x+1))```\r\n\r\n(where x is the original intercept, uniformly distributed between -1 and 1, and d is the dimensionality)\r\n\r\nI've got the derivation here: https://github.com/tcstewar/testing_notebooks/blob/master/Intercept%20Distribution%20.ipynb along with some analysis showing that this makes a pretty big improvement in decoder accuracy for higher dimensions.  Oddly enough, it does make decoding a constant function slightly worse, and it seems to make 2 and 4-dimensional linear functions a very very small amount worse.  The graphs below show the decoder error for various functions (constant (1), linear (x_i), squaring (x_i^2), and other quadratics (x_i * y_i) both with and without this system.  The green bars are with this system.\r\n\r\nFor a constant function, we actually get worse with this method.\r\n![image](https://cloud.githubusercontent.com/assets/2380772/20647394/6c8784f2-b461-11e6-8c84-150abc431138.png)\r\n\r\nFor a linear function, we get a touch worse up until D=4, and then we get better\r\n![image](https://cloud.githubusercontent.com/assets/2380772/20647397/890740f4-b461-11e6-81f9-a8ffa9e37dd4.png)\r\n\r\nFor a squaring function, we get an improvement everywhere\r\n![image](https://cloud.githubusercontent.com/assets/2380772/20647398/a38d6246-b461-11e6-97a9-c9afa7ecc863.png)\r\n\r\nAnd same for the other pairwise quadratics\r\n![image](https://cloud.githubusercontent.com/assets/2380772/20647400/ae5489fc-b461-11e6-96cf-e4cb0517a7cb.png)\r\n\r\nThe implementation of this ends up being fairly compact, but it does rely on scipy:\r\n\r\n```python\r\nclass AreaIntercepts(nengo.dists.Distribution):\r\n    dimensions = nengo.params.NumberParam('dimensions')\r\n    base = nengo.dists.DistributionParam('base')\r\n    \r\n    def __init__(self, dimensions, base=nengo.dists.Uniform(-1, 1)):\r\n        super(AreaIntercepts, self).__init__()\r\n        self.dimensions = dimensions\r\n        self.base = base\r\n        \r\n    def __repr(self):\r\n        return \"AreaIntercepts(dimensions=%r, base=%r)\" % (self.dimensions, self.base)\r\n    \r\n    def transform(self, x):\r\n        sign = 1\r\n        if x > 0:\r\n            x = -x\r\n            sign = -1\r\n        return sign * np.sqrt(1-scipy.special.betaincinv((self.dimensions+1)/2.0, 0.5, x+1))\r\n    \r\n    def sample(self, n, d=None, rng=np.random):\r\n        s = self.base.sample(n=n, d=d, rng=rng)\r\n        for i in range(len(s)):\r\n            s[i] = self.transform(s[i])\r\n        return s\r\n```\r\n\r\nThis feels like something that might help out SPA models a fair bit...."},"comment":{"url":"https://api.github.com/repos/nengo/nengo_extras/issues/comments/264994369","html_url":"https://github.com/nengo/nengo_extras/issues/34#issuecomment-264994369","issue_url":"https://api.github.com/repos/nengo/nengo_extras/issues/34","id":264994369,"user":{"login":"arvoelke","id":5420057,"avatar_url":"https://avatars.githubusercontent.com/u/5420057?v=3","gravatar_id":"","url":"https://api.github.com/users/arvoelke","html_url":"https://github.com/arvoelke","followers_url":"https://api.github.com/users/arvoelke/followers","following_url":"https://api.github.com/users/arvoelke/following{/other_user}","gists_url":"https://api.github.com/users/arvoelke/gists{/gist_id}","starred_url":"https://api.github.com/users/arvoelke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arvoelke/subscriptions","organizations_url":"https://api.github.com/users/arvoelke/orgs","repos_url":"https://api.github.com/users/arvoelke/repos","events_url":"https://api.github.com/users/arvoelke/events{/privacy}","received_events_url":"https://api.github.com/users/arvoelke/received_events","type":"User","site_admin":false},"created_at":"2016-12-05T22:11:45Z","updated_at":"2016-12-05T22:11:45Z","body":"> We also don't have a mathematical proof that the proposed distribution (AreaIntercepts) is the right thing to do.\r\n\r\nCertainly. An intuition that was shared by @celiasmith, @hunse, and I was that the intercepts should be roughly correlated with the magnitude of the derivative of the desired function (projected onto the encoder). For instance, if the function is linear, then the optimal setting for ReLU is to simply be active across the entire space (intercepts at -1). But if there is a sharp change in the function, for instance if it's piecewise linear about 0, then you might want there to be intercepts at 0 as well. Linking this to the intuition from the talk I gave, we want the equivalence classes (the portions of space where the same set of neurons are active, i.e., the regions between the intercepts) to be well-approximated by quadratic monomials for the desired function.\r\n\r\nHowever, I haven't tried to work this out formally within the context of this thread. I'll be revisiting this at some point but thought I'd just state the above now in case it helps. "}}
{"actor":{"display_login":"arvoelke"}}