Originally a summer cottage, today the mansion is a working Relais and Chateaux property known as the Castle Hill Inn. The 19th century mansion sits on a cliff overlooking Narragansett Bay where it meets the ocean. If you're in the area, the restaurant is first rate.
The cottage was built by Alexander Agassiz, a scientist and intellectual, who named it Castle Hill. (More about him: http://xroads.virginia.edu/~ma01/davis/newport/biographies/agassiz.html).