Fernside is considered the oldest vacation house Worcester County and had served as such until 1989. The central section of the house was built in 1835 by Benjamin Harrington on the site of an earlier house built in 1771 by Joseph Eveleth. A subsequent owner, Wilkes Roper added the north and south wings and operated a boarding house for Harvard professors and students. These additions were made in 1870-2. In 1885 F. A. Boyd leased the house and used it as a boarding house. It was during this period that the building was used as a tavern for some time and fell into disrepair.

In 1890 the house was bought by the Working-Girls’ Club in Boston (the first of its kind formed in 1886). The house was set up by donations from wealthy patrons to enable the working class women the opportunity for a summer vacation away from the city. The house was refurbished and kept in excellent condition for many years and was expertly and lovingly managed by Josephine Allen. The Vacation House received its last guest in 1989. However, the new owners, Richard and Jocelyn Morrison have restored Fernside into a bed, breakfast, and conference center.