In 1905 William Crichton (Kenneth More) is the efficient butler in the London household of the Earl of Loam (Cecil Parker) and his family. Crichton knows his place in the highly class-conscious English society. The Earl insists that all men are equal, and to prove it, he orders his daughters to treat the staff as guests during an uncomfortable afternoon tea. Lady Brocklehurst (Martita Hunt) arrives and strongly disapproves of the arrangement, as does Crichton.
When Lady Catherine (Mercy Haystead), one of the Earl's daughters, is arrested at a suffragette protest, Crichton recommends the family take a trip on the Earl's steam yacht to the South Seas until the scandal dies down. When the yacht's motors explode during a storm, all are forced to abandon ship. By the time Crichton rescues the still sleeping "tweeny" maid Eliza (Diane Cilento), the lifeboats have already departed. They jump into the water and are picked up by the wrong boat, the one reserved for the upper class.
Crichton, Eliza, the Earl, his daughters Mary (Sally Ann Howes), Catherine and Agatha (Miranda Connell), clergyman John Treherne (Jack Watling), and Lord Ernest Woolley (Gerald Harper) land on a deserted island. The aristocrats prove to be helpless in their strange new surroundings. It is up to Crichton to start a fire, provide shelter, and find food.
When the abandoned yacht appears and drifts into an offshore rock formation, Crichton swims out to salvage what he can. Upon his return, however, the others order him to pick up unnecessary luxuries rather than vital supplies on his next trip. He reluctantly complies, but at dinner, he insists he must take charge. The Earl instead discharges him. Eliza throws in her lot with Crichton, and the two depart.
The Earl and his party soon realise that they cannot do without Crichton and capitulate, Mary being the sole holdout. She is eventually forced to give in as well.
After two years, the social order has been completely upended: Crichton, now affectionately known as "the Guv", is in charge, while his former betters are his servants. In fact, the aristocrats have toughened up admirably and are quite content with their lot. Romantically, however, the situation is in disarray, as everyone waits to see whether Crichton will choose Mary or "Tweeny" (as Eliza is now called), both of whom are deeply in love with him. All three of the other men are smitten with Tweeny.
Finally, Crichton chooses Mary. However, just as they are exchanging wedding vows, a ship is sighted. Mary begs the others not to light a signal fire, reminding them how happy they have been on the island, but in the end, Crichton does so. When a rescue party lands, he has put on his butler's uniform and resumed his servile duties, much to the discomfort of the others.
The castaways return to London. Woolley writes a book of their experiences, one that portrays him as the saviour of the group. Lady Brocklehurst, suspecting that the work is full of lies, insists on questioning all of the party privately. Crichton tells the truth, but in such a way as to conceal everything. After the Duchess leaves, he tenders his resignation. When the Earl offers financial assistance for his plan to start a business, Crichton shows him a bag of valuable pearls acquired while on the island. Mary begs him to return there with her, but Crichton tells her they cannot fight civilisation. In the end, Tweeny is ecstatic when he accepts her offer to go with him.
What did Earl and his party soon realise?
That they cannot do without Crichton and capitulate, Mary being the sole holdout