Dunyvaig Castle, (Scottish Gaelic: "Dùn Naomhaig" , Anglicised "Fort of the galleys", also known as "Dunnyveg") is located on the south side of Islay, in Argyll, Scotland, on the shore of Lagavulin Bay, 4 km from Port Ellen. The castle was once a naval base of the Lord of the Isles, chiefs of Clan Donald. It was held by the chiefs of the Clan MacDonald of Dunnyveg.

Based on the paragraph above can we conclude that "the castle was not far from port ellen"?
Yes