Williams married his first wife, Valerie Velardi, in June 1978, following a live-in relationship with comedian Elayne Boosler. Velardi and Williams met in 1976 while he was working as a bartender at a tavern in San Francisco.