From the fourteenth to the sixteenth century the area on which Rosehearty presently stands was occupied by a number of crofts, with the buildings clustered together.

Then, in 1573, the JAM was erected. This building was lived in till before the last war. It is now a complete ruin and stands on Union Street facing the Temperance Hotel. A triangular stone plaque is in place above the old gateway.

Another house was built around this time which was called the lodging house. A stone plaque once adorned the archway leading into the courtyard. When the present building, now the Bay Hotel, was built on the site of this dowager lady Pitsligo's residence, the plaque was kept and survives to this day. It bears a rose and heart and the inscription NUNC TROJA VBI SEGES.

By the seventeenth century there were two small settlements and these were joined together as a burgh of barony, granted by Royal Charter, dated thirteenth July, 1681, by King Charles at Windsor Castle. On the 18th October, 1684, a town's charter was signed. Lord Pitsligo built a new seatown for the fisherfolk and this became known as "Newtoon". In later years this was reversed as new buildings emerged on the landward side. These residents were known by the fishers as "Newtooners" and this persisted until recent times; the dividing line then was Ward Road.

The tollbooth built in 1638 was demolished in 1902. The business affairs of the family of Pitsligo were managed in the upper rooms, while the lower part consisted of the town's chambers and a prison cell. It had witnessed considerable changes. Part of it was turned into a cooperage last century, with the upstairs serving as a school. The building extended across the entrance to the Square from Union Street.

The oldest buildings lived in today are the Forbes Arms, built in 1746, and Margaret's Hairdressers on the Square, built in 1748.

Most of the olden buildings are nineteenth century and belong to the town's heyday of the herring fishing.

Much evidence survives to this day of Rosehearty's golden years of the herring industry. The old curing stations here have been converted without being distorted beyond recognition, for modern requirements. Two "Red Herring" buildings and a boatbuilding shed are among some of the landmarks still seen at Rosehearty. Lord Pitsligo's old seatoun is at North Street and although the buildings here have all been modernised, the layout of the gable end to the sea houses stem from this period.