A story is told that at some ancient date two brothers, members of the O’Connell family, occupied Ballycarbery Castle as constables for the MacCarthy Mór, lords of much of south Kerry. The elder sibling lived on the lower storey, the younger on the upper and both wished to offer dinner to their lord on the same night. Accordingly to settle the dispute MacCarthy Mór declared he would eat with whichever brother had the meal prepared first. The elder then ordered his servants to block up all access to the upper floors and stand guard so that nobody could enter or exit. When his younger brother discovered this scheme, he arranged to have all his pots filled with Spanish wine in which the food was cooked, and by this means he had dinner ready first and was able to entertain the MacCarthy Mór.

The present Ballycarbery Castle appears to be of 16th century origin, although built on the site of an older fortified structure. From about 1350 the building was occupied and under the care of the O’Connells, serving their overlord the aforementioned MacCarthy Mór. The main body of the building is substantial, measuring some 74 by 42.6 feet with a projecting tower in one corner that rising four storeys. Sections of a surrounding bawn wall remain. A large vaulted chamber on the ground floor survives, and portions of other rooms at this level: much the same is true of the floor above, accessed by one of the building’s two staircases. It is easy to understand why a castle was erected here, since the spot on which it stands is close to the edge of the Atlantic, with views for several miles south-west towards Valencia Island and beyond, and towards Caherciveen and its hinterland to the east. Whoever held the castle could see the approach of any potential opponent, on either land or sea, well in advance.

Ballycarbery Castle appears to have remained in the custody of the O’Connells until the early 17th century: by this time, a large portion of the MacCarthy Mór territory – including the Lakes of Killarney – had passed into the possession of the Brownes, future Earls of Kenmare. The castle itself seems to have been attacked and badly damaged by English troops in 1652 but enough survived to ensure its survival. In the 18th century a family called Lauder built a new house attached to one side of the bawn wall: it appears in a watercolour made in 1792 by Daniel Grose. Already a ruin when he saw it, this building was demolished at the start of the last century. Grose’s picture shows large chunks of masonry fallen from the southern section of the castle, this damage presumably from the mid-17th century when the building was subject to attack. In 1910 it was noted that a tenant farmer had demolished some twenty-five feet of the southern outer wall and was clearing away quantities of stone work until cautioned to desist. It looks as though little has changed since then.