Robert Clary was born Robert Max Widerman in Paris, France, on March 1, 1926. He was one of 14 children in his Orthodox Jewish family. Clary was only 16 when he was imprisoned in a concentration camp at Ottmuth, Poland (per the Florida Sun-Sentinel), and then in Buchenwald. The actor credited his youth, good health, and ability to sing and play music — he had been performing on the radio since he was 12 years old and performed for SS members every other Sunday at Buchenwald — with his survival. 

Clary was initially reluctant to talk of his experiences at Buchenwald but eventually began taking on speaking engagements, traveling the world, and telling his story. "For 36 years I kept these experiences during the war locked up inside myself. But those who are attempting to deny the Holocaust, my suffering and the suffering of millions of others have forced me to speak out," Clary once said, per The Hollywood Reporter. He subsequently wrote a memoir about his life, "From the Holocaust to Hogan's Heroes: The Autobiography of Robert Clary." After 31 months of imprisonment, Clary was liberated with the rest of the camp's surviving prisoners on April 11, 1945.

After returning to Paris to become a nightclub singer, he attracted the attention of Capitol Records, who brought him to Los Angeles to record an album. While in Los Angeles, Clary found himself being mentored by legendary Broadway and Vaudeville performer Eddie Cantor, who helped launched his career as a nightclub singer and Broadway star. Through his mentor, he met Natalie Cantor, one of the actor's five daughters. Cantor and Clary would remain friends for over a decade before marrying. Their union would last for 32 years, until her death in 1997.