{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/1652","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/1652/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/1652/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/1652/events","html_url":"https://github.com/fchollet/keras/issues/1652","id":131812803,"number":1652,"title":"Need help on the implementation of the 2D-RNN","user":{"login":"shampool","id":13158661,"avatar_url":"https://avatars.githubusercontent.com/u/13158661?v=3","gravatar_id":"","url":"https://api.github.com/users/shampool","html_url":"https://github.com/shampool","followers_url":"https://api.github.com/users/shampool/followers","following_url":"https://api.github.com/users/shampool/following{/other_user}","gists_url":"https://api.github.com/users/shampool/gists{/gist_id}","starred_url":"https://api.github.com/users/shampool/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shampool/subscriptions","organizations_url":"https://api.github.com/users/shampool/orgs","repos_url":"https://api.github.com/users/shampool/repos","events_url":"https://api.github.com/users/shampool/events{/privacy}","received_events_url":"https://api.github.com/users/shampool/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-02-06T03:31:24Z","updated_at":"2016-02-07T15:40:28Z","closed_at":"2016-02-07T09:04:39Z","body":"The current idea is, map the original (Nsample, timestep ,input_dim) to a grid  (Nsample, grid_shape[0], grid_shape[1] ,input_dim).\r\nThen apply nested scan to scan column by column. \r\nAt each step, updates the current state and activation based on the row and column predecessors.\r\nSo the initial_states is [row_states] + [col_states].  \r\n \r\n```\r\n\r\nclass MultiDimRNN(Recurrent):\r\n    '''grid_shape: define the 2D shape of the grids. Eg. there are in total 5*6 grids'''\r\n\r\n    \r\n    def __init__(self, output_dim, grid_shape , \r\n                 init='glorot_uniform', inner_init='orthogonal',\r\n                 activation='sigmoid', inner_activation='hard_sigmoid',\r\n                 **kwargs):\r\n        \r\n        self.output_dim = output_dim\r\n        self.grid_shape = tuple(grid_shape)\r\n  \r\n        self.init = initializations.get(init)\r\n        self.inner_init = initializations.get(inner_init)\r\n        self.activation = activations.get(activation)\r\n        self.inner_activation = activations.get(inner_activation)\r\n        super(MultiDimRNN, self).__init__(**kwargs)\r\n\r\n    def build(self):\r\n        input_shape = self.input_shape\r\n        if self.stateful:\r\n            self.reset_states()\r\n        else:\r\n            # initial states: all-zero tensor of shape (output_dim)\r\n            self.states = [None for _ in range(np.sum(self.grid_shape))]\r\n            #for state_id in range(np.sum(self.grid_shape)):\r\n            #    self.states.append(K.zeros((input_shape[0], self.output_dim)) )\r\n\r\n        input_dim = input_shape[2] #Nsample*timestep*input_dim\r\n        self.input_dim = input_dim\r\n\r\n        self.W = self.init((input_dim, self.output_dim))\r\n        self.U_row = self.inner_init((self.output_dim, self.output_dim))\r\n        self.U_col = self.inner_init((self.output_dim, self.output_dim))\r\n        self.b = K.zeros((self.output_dim,))\r\n        self.params = [self.W, self.U_row, self.U_col, self.b]\r\n\r\n        if self.initial_weights is not None:\r\n            self.set_weights(self.initial_weights)\r\n            del self.initial_weights\r\n            \r\n    def get_initial_states(self, X):\r\n        # build an all-zero tensor of shape (samples, output_dim)\r\n        initial_state = K.zeros_like(X)  # (samples, timesteps, input_dim)\r\n        initial_state = K.sum(initial_state, axis=1)  # (samples, input_dim)\r\n        reducer = K.zeros((self.input_dim, self.output_dim))\r\n        initial_state = K.dot(initial_state, reducer)  # (samples, output_dim)\r\n        initial_states = [initial_state for _ in range(len(self.states))]\r\n        return initial_states\r\n        \r\n    def reset_states(self):\r\n        assert self.stateful, 'Layer must be stateful.'\r\n        input_shape = self.input_shape\r\n        if not input_shape[0]:\r\n            raise Exception('If a RNN is stateful, a complete ' +\r\n                            'input_shape must be provided ' +\r\n                            '(including batch size).')\r\n        if hasattr(self, 'states'):\r\n            for state_id in range(np.sum(self.grid_shape)):  \r\n                K.set_value(self.states[state_id],\r\n                            np.zeros((input_shape[0], self.output_dim)))\r\n        else:\r\n            self.states = []\r\n            for state_id in range(np.sum(self.grid_shape)):\r\n                self.states.append(K.zeros((input_shape[0], self.output_dim)))\r\n\r\n    def step(self, x, row_state, col_state):\r\n        # states only contains the previous output.\r\n        #assert len(states) == 1\r\n        #prev_output = states[0]\r\n        h = K.dot(x, self.W) + self.b\r\n        output = self.activation(h + (K.dot(row_state, self.U_row) + K.dot(col_state, self.U_col))/2 )\r\n        return output, [output]\r\n\r\n    def get_output(self, train=False):\r\n        # input shape: (nb_samples, time (padded with zeros), input_dim)\r\n        X = self.get_input(train)\r\n        assert K.ndim(X) == 3\r\n        # get input, then map it to the grid_shape\r\n        inner_data = K.reshape(X, (-1,) + self.grid_shape + (self.input_shape[2],))  # for example 16*16 *900 (16 * 16 grids and each contains 900 pixels)\r\n       \r\n        \r\n        if K._BACKEND == 'tensorflow':\r\n            if not self.input_shape[1]:\r\n                raise Exception('When using TensorFlow, you should define ' +\r\n                                'explicitly the number of timesteps of ' +\r\n                                'your sequences. Make sure the first layer ' +\r\n                                'has a \"batch_input_shape\" argument ' +\r\n                                'including the samples axis.')\r\n\r\n        mask = self.get_output_mask(train)\r\n\r\n        \r\n\r\n        if mask:\r\n            # apply mask\r\n            X *= K.cast(K.expand_dims(mask), X.dtype)\r\n            masking = True\r\n        else:\r\n            masking = False\r\n     \r\n        #grid_mask = K.reshape(mask, (-1,) +  self.grid_shape)\r\n        \r\n        if self.stateful:\r\n            initial_states = self.states\r\n        else:\r\n            initial_states = self.get_initial_states(X) #build an all-zero tensor of shape (samples, output_dim)\r\n        \r\n        \r\n        \r\n\r\n        last_output, outputs, states = self.rnn_grid(self.step, inner_data, self.grid_shape,initial_states,\r\n                                             go_backwards=self.go_backwards,\r\n                                             masking=masking)\r\n        \r\n        if self.stateful:\r\n            self.updates = []\r\n            for i in range(len(states)):\r\n                self.updates.append((self.states[i], states[i]))\r\n        \r\n        if self.return_sequences:\r\n            return K.reshape(outputs, (-1,) + (np.prod(self.grid_shape), self.input_shape[2]))\r\n        else:\r\n            return last_output\r\n\r\n    def rnn_grid(self ,step_function, inputs, grid_shape,initial_states,\r\n        go_backwards=False, masking=True):\r\n        row_states = initial_states[0:grid_shape[0]]\r\n        col_states = initial_states[grid_shape[0]:]\r\n\r\n        inputs = inputs.dimshuffle((2,1, 0, 3))  # col, row, nsample, dim\r\n        #inputs = input.dimshuffle(tuple(range(1,len(grid_shape)+1))+(0, len(grid_shape)+1) )\r\n                \r\n        def _step(input,row, row_state, col_state): #[row_states] + [col_state]\r\n            #'''it needs the left, and upper grid as it's predecessor,  \r\n\t\t\t#   it only update the col_state after process one local patch.\r\n\t\t\t#   although it needs two states to compute each step, but it actually only updates the upper column state, for the\r\n\t\t\t#   left row_state, it only use it.\r\n\t\t\t#'''\r\n            states = [col_state]\r\n            output, new_states = step_function(input, row_state, states[0])  # this function actually should only return one state\r\n\r\n            if masking:\r\n                # if all-zero input timestep, return\r\n                # all-zero output and unchanged states\r\n                switch = T.any(input, axis=-1, keepdims=True)\r\n                output = T.switch(switch, output, 0. * output)\r\n                return_states = []\r\n                for state, new_state in zip(states, new_states):\r\n                    return_states.append(T.switch(switch, new_state, state))\r\n                return [output] + return_states\r\n            else:\r\n                return [output] + new_states\r\n\r\n             \r\n        def loop_over_col(coldata, col_ind,col_state, rows_states, rows_model):\r\n            #'''when loops over a column, it needs the left column as row_states,\r\n\t\t\t#row_states, col_states are already concatenated to be tensors for scan to process before call this function.\r\n\t\t\t#it return the results as well as this column response as the row_states to next column.\r\n\t\t\t#it should return the last column_state\r\n\t\t\t#'''\r\n            results , _ = theano.scan( fn = _step,\r\n                                       sequences = [coldata, rows_model, rows_states], \t\t\t\t\t\t\t\t\t   \r\n                                       outputs_info=[None] +  [col_state] \r\n                                       )\r\n\r\n            new_row_states = results[1:] #'''list of tensor of row_size * nsample * out_dim, but we need to modify it to be list'''\r\n            col_vals = results[0]  \r\n\t\t   #'''tensor of row_size * nsample * out_dim'''\r\n\t\t   #'''the length is the number of states for each single actual step'''\r\n            single_state_num = len(new_row_states) \r\n            returned_row_states = []\r\n            for timestep_id in range(len(rows_model)):\t\t\r\n\t\t\t    #'''new_row_states is a list of tensor of size row_size * nsample * out_dim\r\n\t\t\t\t#   returned states should be [timestep_1_state_1,...,timestep_1_state_n, ...., timestep_m_state_1,...,timestep_m_state_n],\r\n\t\t\t\t#   n is the number of required states for each single actual step function, most time it is 1. but CWRNN has 2.\r\n\t\t\t\t#   m is the row_size, = grid_shape[0]\r\n\t\t\t\t#'''\r\n\t\t\t   for state_id in range(single_state_num):\r\n\t\t\t\t    returned_row_states.append(T.squeeze(new_row_states[state_id][timestep_id]))\r\n            returned_col_state = []\r\n            for state_id in range(single_state_num):\r\n\t\t\t  returned_col_state.append(T.squeeze(new_row_states[state_id][-1]))\r\n            #'''output should be list corresponding to the outputs_info'''\r\n            return [col_vals] + T.concatenate(returned_row_states, axis =0), T.stack(returned_col_state, axis =0)\r\n\t\t\t\r\n\r\n        inputs_model = dict(input = inputs, taps=[0])\r\n        rows_model = T.arange(self.grid_shape[0])\r\n        cols_model = T.arange(self.grid_shape[1])\r\n\r\n        #'''loop_over_col(coldata, col_ind, rows_states, col_states, rows_model)'''\r\n\t\t#'''Maybe I need to concatenate all states list to tensor at this stage, scan deal with list of tensor variable, not\r\n\t\t#list of list.'''\r\n        results, _ = theano.scan( fn = loop_over_col,\r\n                                    sequences = [inputs_model, cols_model, T.stack(col_states, axis =0)],\r\n\t\t\t\t\t\t\t\t#'''return grid_results, row_states, col_state'''\r\n                                    outputs_info=[None, T.stack(row_states, axis =0), None],\r\n                                    non_sequences = [rows_model] \r\n            )\r\n        #update col_states here\r\n\t\t#'''results are list of [grid_tensor, row_state_tensor, col_state]'''\r\n        outputs =  results[0]  #'''tensor type'''\r\n        row_states_tensor  = results[1][-1]  #'''we only need the last column as the row_states, tensor type'''\r\n        col_states_tensor  = results[2]  #'''tensor type'''\r\n        \r\n        \r\n        # deal with Theano API inconsistency\r\n        #if type(results) is list:\r\n        #    outputs = results[0]\r\n        #    states = results[1:]\r\n        #else:\r\n        #    outputs = results\r\n        #    states = []\r\n        outputs = T.squeeze(outputs)\r\n        last_output = outputs[-1,-1]\r\n\r\n        #outputs  col, row, nsample, dim\r\n        outputs = outputs.dimshuffle((2,1,0,3))\r\n\r\n        row_states = [T.squeeze(state[-1]) for state in row_states_tensor]\r\n        col_states = [T.squeeze(state[-1]) for state in col_states_tensor]\r\n        return last_output, outputs, row_states + col_states\r\n        \r\n\r\n    def get_config(self):\r\n        config = {\"grid_shape\": self.grid_shape,\r\n                  \"output_dim\": self.output_dim,\r\n                  \"init\": self.init.__name__,\r\n                  \"inner_init\": self.inner_init.__name__,\r\n                  \"activation\": self.activation.__name__}\r\n        base_config = super(MultiDimRNN, self).get_config()\r\n        return dict(list(base_config.items()) + list(config.items()))\r\n\r\n            \r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/181034919","html_url":"https://github.com/fchollet/keras/issues/1652#issuecomment-181034919","issue_url":"https://api.github.com/repos/fchollet/keras/issues/1652","id":181034919,"user":{"login":"WeidiXie","id":9581238,"avatar_url":"https://avatars.githubusercontent.com/u/9581238?v=3","gravatar_id":"","url":"https://api.github.com/users/WeidiXie","html_url":"https://github.com/WeidiXie","followers_url":"https://api.github.com/users/WeidiXie/followers","following_url":"https://api.github.com/users/WeidiXie/following{/other_user}","gists_url":"https://api.github.com/users/WeidiXie/gists{/gist_id}","starred_url":"https://api.github.com/users/WeidiXie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/WeidiXie/subscriptions","organizations_url":"https://api.github.com/users/WeidiXie/orgs","repos_url":"https://api.github.com/users/WeidiXie/repos","events_url":"https://api.github.com/users/WeidiXie/events{/privacy}","received_events_url":"https://api.github.com/users/WeidiXie/received_events","type":"User","site_admin":false},"created_at":"2016-02-07T15:40:27Z","updated_at":"2016-02-07T15:40:27Z","body":"I'm currently not by the pc,  and will go to the lab later, maybe we can debug it together. I need it as well.  Could you drop me an email."}}