disclosed is a signal detection method for a receiver of a multiple - input multiple - output communication system having n t and n r transmission and reception antennas , respectively , and an n t × n r channel matrix . the signal detection method includes selecting a plurality of sub - channel matrices having smaller dimensions than the n t × n r channel matrix , estimating transmission symbols received from corresponding transmission antennas using a maximum likelihood decoding method with respect to a received signal corresponding to each of the sub - channel matrices , estimating transmission symbols received from the remaining transmission antenna using the estimated transmission symbols , temporarily storing a candidate symbol set composed of the estimated transmission symbols , and comparing candidate symbol sets corresponding to all the sub - channel matrices and determining the smallest candidate symbol set as a transmission symbol set .