Milt Miller grew up in Detroit where his dad, Denver, worked for 43 years on the Wabash Railroad. His mom, Esther, was housewife, but she worked raising Milt, his older half-brother, Warren Sessler, and sisters, Grace, a retired nurse, and Marilyn, also a housewife.