Port Townsend is noted for a vast collection of Victorian homes and significant historical buildings. The city has more than a dozen larger buildings that are well preserved, including the Carnegie Library, the Federal Building (now commonly known simply as the city's post office), the Rose Theatre and the Elks Lodge, which now houses Silverwater Cafe. There is also Fort Worden with its pre-World War I architecture including the publicly accessible Olympic Youth Hostel. Perhaps the most stunning historical structure in Port Townsend is the Jefferson County Courthouse, a spectacular representation of the Richardsonian Romanesque architectural style.
In 1976, the Downtown waterfront and parts of Uptown were designated a Registered Historic District. Later, Fort Worden (now part ofFort Worden State Park) and the City of Port Townsend were designated National Historic Landmarks. The city is further recognized as one of only three Victorian seaports on the National Register of Historic Places.

The Bell Tower on the bluff above downtown is one of two known towers of this type in the United States. It was used from 1890 to the 1940s to call volunteer firefighters. It was restored in 2003 by the Jefferson County Historical Society. The second bell tower is located in Helena, Montana, and was also used to summon volunteers and alert townspeople of any fires during late 19th century early gold rush days.