a transformation coding apparatus for transforming blocks of data corresponding to a boundary of an object . the apparatus includes a transformer for transforming an input block and producing transformation coefficients , a transformation coefficient selector for selecting and outputting certain transformation coefficients based on the transformation coefficients and the input block , an inverse transformer for inversely transforming the transformation coefficients selected by the transformation coefficient select means , and outputting a restored block , and an encoder for variable - length - coding the select coefficients selected by the transformation coefficient select means . a repetition controller calculates a mean square error of the input data block and the restored data block , compares , the calculated mean square error with a predetermined reference value , and determines whether to continue selecting coefficients . the transformation coding apparatus selects the coefficients which maximize the ratio of the mean square error δmse , to thereby enhance the quality of the picture and reduce the amount of necessary data bits .