{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/3842","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/3842/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/3842/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/3842/events","html_url":"https://github.com/fchollet/keras/issues/3842","id":178479761,"number":3842,"title":"`activations.softexp()`","user":{"login":"hobson","id":292855,"avatar_url":"https://avatars.githubusercontent.com/u/292855?v=3","gravatar_id":"","url":"https://api.github.com/users/hobson","html_url":"https://github.com/hobson","followers_url":"https://api.github.com/users/hobson/followers","following_url":"https://api.github.com/users/hobson/following{/other_user}","gists_url":"https://api.github.com/users/hobson/gists{/gist_id}","starred_url":"https://api.github.com/users/hobson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hobson/subscriptions","organizations_url":"https://api.github.com/users/hobson/orgs","repos_url":"https://api.github.com/users/hobson/repos","events_url":"https://api.github.com/users/hobson/events{/privacy}","received_events_url":"https://api.github.com/users/hobson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-09-21T22:38:53Z","updated_at":"2016-09-22T19:32:37Z","closed_at":null,"body":"[This paper](https://arxiv.org/pdf/1602.01321.pdf) explains the numerous advantages of a  _soft exponential_ activation function including improved training, simplified neural nets, and improved accuracy for models that rely on multiplicative transforms of the input, such as dot product, cross product, and polynomial transforms.\r\n\r\nMy draft implementation is:\r\n\r\n```python\r\ndef softexp(x, alpha=0., max_value=None):\r\n    \"\"\"Soft Exponential activation function by Godfrey and Gashler\r\n\r\n    See: https://arxiv.org/pdf/1602.01321.pdf\r\n\r\n    α == 0:  f(α, x) = x\r\n    α  > 0:  f(α, x) = (exp(αx)-1) / α + α\r\n    α  < 0:  f(α, x) = -ln(1-α(x + α)) / α\r\n    \"\"\"\r\n    if alpha == 0:\r\n        return x\r\n    elif alpha > 0:\r\n        return alpha + (K.exp(alpha * x) - 1.) / alpha\r\n    else:\r\n        return - K.log(1 - alpha * (x + alpha)) / alpha\r\n```\r\n\r\nI'm testing it to see that it matches these plots by the paper's authors and improves my model performance.\r\n\r\n![image](https://cloud.githubusercontent.com/assets/292855/18731374/62fc7a44-8010-11e6-8993-0709348d9960.png)\r\n\r\nHere's the fork and branch: https://github.com/totalgood/keras/tree/softexp"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/249004971","html_url":"https://github.com/fchollet/keras/issues/3842#issuecomment-249004971","issue_url":"https://api.github.com/repos/fchollet/keras/issues/3842","id":249004971,"user":{"login":"hobson","id":292855,"avatar_url":"https://avatars.githubusercontent.com/u/292855?v=3","gravatar_id":"","url":"https://api.github.com/users/hobson","html_url":"https://github.com/hobson","followers_url":"https://api.github.com/users/hobson/followers","following_url":"https://api.github.com/users/hobson/following{/other_user}","gists_url":"https://api.github.com/users/hobson/gists{/gist_id}","starred_url":"https://api.github.com/users/hobson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hobson/subscriptions","organizations_url":"https://api.github.com/users/hobson/orgs","repos_url":"https://api.github.com/users/hobson/repos","events_url":"https://api.github.com/users/hobson/events{/privacy}","received_events_url":"https://api.github.com/users/hobson/received_events","type":"User","site_admin":false},"created_at":"2016-09-22T19:32:37Z","updated_at":"2016-09-22T19:32:37Z","body":"I attempted to follow the ParametricSoftPlus pattern. But I cannot figure out how to use `K.switch(alpha,...` when `alpha` is trainable (presumably a Tensor), but `switch()` requires a scalar bool as the first arg.  Will send PR so you can comment on lines directly.\r\n\r\n```python\r\nclass ParametricSoftExp(Layer):\r\n    \"\"\"Soft Exponential activation function with trainable alpha\r\n\r\n    See: https://arxiv.org/pdf/1602.01321.pdf by Godfrey and Gashler\r\n\r\n    # Input shape\r\n        Arbitrary. Use the keyword argument `input_shape`\r\n        (tuple of integers, does not include the samples axis)\r\n        when using this layer as the first layer in a model.\r\n\r\n    # Output shape\r\n        Same shape as the input.\r\n\r\n    # Arguments\r\n        alpha_init: float. Initial value of the alpha weights.\r\n        weights: initial alpha weights, as a list of 1 numpy array.\r\n                 if both weights & alpha_init are provided, weights overrides alpha_init\r\n\r\n    Soft Exponential f(α, x):\r\n        α == 0:  x\r\n        α  > 0:  (exp(αx)-1) / α + α\r\n        α  < 0:  -ln(1-α(x + α)) / α\r\n    \"\"\"\r\n    def __init__(self, alpha_init=0.1,\r\n                 weights=None, **kwargs):\r\n        self.supports_masking = True\r\n        self.alpha_init = K.cast_to_floatx(alpha_init)\r\n        self.initial_weights = weights\r\n        super(ParametricSoftplus, self).__init__(**kwargs)\r\n\r\n    def build(self, input_shape):\r\n        input_shape = input_shape[1:]\r\n        self.alphas = K.variable(self.alpha_init * np.ones(input_shape),\r\n                                 name='{}_alphas'.format(self.name))\r\n        self.trainable_weights = [self.alphas]\r\n\r\n        if self.initial_weights is not None:\r\n            self.set_weights(self.initial_weights)\r\n            del self.initial_weights  # should this be set to None rather than deleted?\r\n\r\n    def call_alpha_gt0(self, x, alpha):\r\n        return alpha + (K.exp(alpha * x) - 1.) / alpha\r\n\r\n    def call_alpha_lt0(self, x, alpha):\r\n        return - K.log(1 - alpha * (x + alpha)) / alpha\r\n\r\n    def call(self, x, mask=None):\r\n        return K.switch(self.alphas > 0, self.call_alpha_gt0,\r\n                        K.switch(self.alphas < 0, self.call_alpha_lt0, x))\r\n\r\n    def get_config(self):\r\n        config = {'alpha_init': float(self.alpha_init)}\r\n        base_config = super(ParametricSoftplus, self).get_config()\r\n        return dict(list(base_config.items()) + list(config.items()))\r\n```"}}
{"actor":{"display_login":"hobson"}}