The EF-4s performed well, so much so that Milwaukee management soon desired to utilize the two EP-4s exclusively on freights. This was being done by 1956, when the passenger Joes were replaced by newly migrated EP-2 Bipolars. After the latter turned out to be ill-suited to the Rocky Mountain Division, they and the EP-4s were replaced by three-unit consists of EMD E-unit and/or FP7 diesels which hauled the "Olympian Hiawatha" end-to-end, unassisted, until its discontinuance in 1961. Neither EP-4 received the post-1955 Union Pacific-inspired Armour Yellow, red and gray paint scheme used on Milwaukee passenger power, such as the Bipolars and GE Passenger Motors.