a semiconductor memory device includes a plurality of word decoders arranged in a plurality of columns , a plurality of word line selecting shift registers corresponding to the respective word decoders to indicate a word line subjected to refresh operation , and a shift control signal generating circuit operable to supply a shift control signal indicative of timing of shift operations to the plurality of word line selecting shift registers , wherein the said shift control signal generating circuit is configured to supply the shift control signal only to a column currently subjected to refresh operation among the plurality of columns .