Rathbone was born in Singapore, the son of American parents Randee Lynn (née Brauner) and Monroe Jackson Rathbone IV. His great-grandfather, Monroe Jackson Rathbone II, was the chairman of Standard Oil of New Jersey, which later became Exxon; Rathbone is distantly related to Civil War general Stonewall Jackson and British actor Basil Rathbone. Because of his father's job at Mobil Oil, Rathbone has lived in places ranging from Indonesia to Midland, Texas. He attended Trinity School of Midland, a college preparatory school, and started out in local theater in Midland with the youth actors program "The Pickwick Players", initially doing musical theatre. For his junior and senior years of high school, Rathbone attended the Interlochen Arts Academy, a private school for the arts in Michigan, where he majored in acting. After graduation, he had planned to go to the Royal Scottish Academy of Music and Drama, but went instead to Los Angeles to experiment with film acting.