The 1860s were a very different time.

It was an era in which you could move to Madison, say, “Hello, I am new in town and I am going to build a castle here,” and your neighbors would shrug and say, “Sure, dude, go for it.”

And that’s exactly what happened.

In 1861, a wealthy Englishman named Benjamin Walker arrived in Madison with his family in tow and began a new life in Wisconsin’s burgeoning capital city. Described as a mysterious, reclusive figure, nobody ever quite knew what brought Walker across the Atlantic, and his dark, brooding demeanor earned him few friends in his new home. What he lacked in company, however, he made up for in architectural ambition.

The year after his arrival, Walker began construction on a permanent estate for him and his family. He purchased a plot of land on the east side of Lake Mendota and built himself a personal castle, drawing inspiration from the Norman fortifications of his native England.

And it was an impressive castle indeed. The imposing sandstone towers of Walker’s new home sat sternly on the shores of Lake Mendota, likely doing little to dissuade his reputation as a mysterious European nobleman. The interior is said to have been saturated in golden wallpaper, plush crimson upholstery and marble fireplaces, with an underground tunnel connecting the fortress to the adjacent barn.

Walker, his wife and his three children lived comfortably in their fortified homestead, but their residence wouldn’t last long.

Dissatisfied with Madison, Walker uprooted his family in 1866 and returned to England, perhaps sick of receiving trick-or-treaters year round at his Gothic lair. He sold the castle to a doctor by the name of Thompson, who soon found the house to be cold, damp and difficult to heat. The estate would change hands several times in the coming years, as few owners could stand to manage the property for long. Unsurprisingly, none of the subsequent owners were adequately prepared to take care of the stronghold, and it gradually fell into a state of disrepair.

Local lore grew about the castle like vegetation over its sandstone battlements, the solemn ruin soon becoming a hotspot for adventurous local children. A Madison old-timer recalls exploring the abandoned rooms within the gothic monument and the countless legends of restless phantoms that patrolled its hallways. University fraternities of the era evidently took a shine to the derelict fortress, requiring pledges to walk alone though the gloomy underground tunnel as a nighttime initiation process.

Unfortunately, the castle’s career as an abandoned attraction was short-lived. It was demolished in the early 1890s, its sandstone walls cannibalized and used in the construction of the present home at 137 E. Gorham St. The estate was sold off and was soon integrated into the surrounding residential neighborhood that had been growing around it. Hardly anything remains today of the grandiose manor, though a small side street perpendicular to East Gorham named Castle Place now runs adjacent to the site of the lakeside fortress.

Few historical records remain that contain details of Walker or his castle, the estate having largely sunk into the gloomy ether of Madison annals save for a few scattered newspaper clippings. To those who lived in Madison during the stronghold’s brief existence, it stood out as a foreboding, near-mythic installation housing one of Madison’s most eccentric and mysterious residents. For the detailed memories of a Madisonian who was a young girl and friends with one of the young Walker daughters, you can read the 1933 memoir written by her granddaughter here. It has been a great help in reconstructing the story of this forgotten Madison artifact.