The good relations Hart established with the imperial authorities in Peking while deputising for Lay, and conflict between Lay and Prince Gong and the Zongli Yamen over the Osborn Fleet, led them to dismiss the difficult and haughty Lay upon his return from leave. Hart was appointed in his place in November 1863, with British approval. As Inspector-General of China's Imperial Maritime Custom Service (IMCS), Hart's main responsibilities included collecting custom duties for the Chinese government, as well as expanding the new system to more sea and river ports and some inland frontiers, standardising its operations, and insisting on high standards of efficiency and honesty. The top echelon of the service was recruited from all the nations trading with China. Hart's advice led to the improvement of China's port and navigation facilities.