The swimmer, thought to be in his late 40s and wearing board shorts, was seen by the coastguard entering the sea at Caswell Bay beach at about 14:30 BST on Wednesday.
He swam breaststroke towards Brandy Cove, but was not seen leaving the water.
Coastguards searched the area but could not find the man. South Wales Police said no one had reported a man missing.
Officers are appealing for information.
A search was launched after a man was seen swimming out to sea in the Gower.