James Stafford came to Paintsville from Virginia in the early 1800s and was a large land owner. He later sold his farm to his son John, and then moved up river to Lewis County. John Stafford (1804-1869) was also a prominent figure who helped establish the city of Paintsville in 1834 and Johnson County as a magistrate in 1843. He also built the Stafford House in 1843, which still stands today in Paintsville as the oldest house in Johnson County.