He was born in Pensacola, Florida. His father, Joseph Angelo D'Allesandro II, was an Italian-American sailor, and his mother was 16-year-old Thelma Testman. By the time Dallesandro was five, his mother was serving five years in a Federal Penitentiary for interstate auto theft. His parents divorced soon afterward. Dallesandro and his brother, Bobby, were taken to New York with their father, who worked as an electrical engineer. Both boys were eventually placed into the Angel Guardian Home in Harlem, prior to being fostered by a couple in Brooklyn. The senior Dallesandro would visit them about once a month at their foster parents' home.
Dallesandro attended a Catholic school until second grade. He and his brother lived with the family until they ran away and were removed from the family by social services. At the age of fourteen Dallesandro and his brother moved to Queens to live with their paternal grandparents. He was kicked out of school for punching the principal, who had insulted his father.
As a teenager, Dallesandro supported himself by prostitution and later nude modeling, appearing most notably in short films and magazine photos for Bob Mizer's Athletic Model Guild. Dallesandro also appeared in at least one gay pornography film. In a later interview, Dallesandro said: "My hustling days were more about trying to take care of myself. Having met those people kind of calmed me down. They showed me a different part of life. My attitude was that it widened my life experience... I realized later that I was looking for a father figure and someone to love me." The street-wise young hustler 'Ned' who appears in Martin Duberman's memoir Cures has been assumed to be Dallesandro.