a maximum likelihood detection apparatus and method in a wireless communication system are provided , in which a euclidean distance are calculated using at least one of channel matrix information , noise power information and modulation order information of each stream , pairwise error rate are calculated using the euclidean distance , an error probability of the each stream is calculated using the per , the error probability of the each stream is ordered and sorted , and ml detection is performed using a result of the ordering and sorting .