Charlie grew up in the Bushwick neighborhood of Brooklyn, NY, the eldest son of Lillian, a telephone operator, and Charles Edward Murphy, who was a transit police officer and amateur comedian. His parents broke up when Charlie was only five, and his father was later murdered by a girlfriend, leaving Charlie as the man of the house when he was only 10. His mother would later remarry, but Charlie and his younger brother endured some struggles, particularly when Lillian went into the hospital to be treated for tuberculosis. For a year or so when Charlie was six and Eddie was four, the brothers lived in a foster home while she was hospitalized. Eddie recalled that foster mom "Mrs. Jenkins" didn't treat Charlie particularly well. "She beat on Charlie. She didn't beat on me, I was too little," he explained.

Despite the tough times in his childhood, "Charlie-Wallie" managed to have a little fun along the way. When he was nine, Murphy got to play a small part in the 1970 film The Landlord, directed by Hal Ashby. In his scene, Murphy and another child are caught stealing hubcaps by the lead character, played by Beau Bridges.