Depends on the state, but federal constitutional protection of voting rights for adult men didn't exist until the 14th Amendment in 1868. Until then, voting qualifications were a state matter, and even after that, the "penalty" was simply a proportional reduction in representation for the state.