Ruffin's father was strict and at times violently abusive. Ruffin's mother died ten months after his birth in 1941; and his father married Earline, a schoolteacher, in 1942. As a young child, Ruffin, along with his other siblings (older brothers Quincy and Jimmy, and sister Reada Mae) traveled with their father and their stepmother as a family gospel group, opening shows for Mahalia Jackson and The Five Blind Boys of Mississippi, among others. Ruffin sang in the choir at Mount Salem Methodist Church, talent shows and wherever else he could. In 1955, at the age of 14, he left home under the guardianship of a minister, Eddie Bush, and went to Memphis, Tennessee, with the purpose of pursuing the ministry.