David Olusoga traces the house's fortunes in the 18th and 19th centuries. He begins by investigating the death of owner Joseph Holbrook, and discovering why his daughter was written out of the will. A later resident of the home rises to become mayor in the 1830s, but had to find a way to prevent riots that had ravaged the city years earlier from occurring again. David also delves into the life of a servant who worked in the house in the 1870s, only to leave domestic servitude for an abusive marriage, and discovers how a successful teacher who later owned the house ended up committed to an asylum.