On December 31, 2013 (New Years' Eve), Avery died aged 68, from complications following open heart surgery at Glendale Memorial Medical Center. He was survived by his mother, Florence Avery (1924–2016), as well as his wife and stepson.
Janet Hubert, who portrayed his original on-screen wife Vivian on The Fresh Prince of Bel-Air, said after his death, "RIP James, all the world is a stage, and we are all merely players in this production called LIFE." Will Smith commented on Avery's death by saying, "Some of my greatest lessons in Acting, Living and being a respectable human being came through James Avery. Every young man needs an Uncle Phil. Rest in Peace." Joseph Marcell called Avery a "gentle giant".
The song "No Role Modelz" from 2014 Forest Hills Drive by J. Cole pays tribute to Avery. It contains the lyric "First things first rest in peace Uncle Phil. For real, you the only father that I ever knew."
Avery's remains were later cremated and scattered near the Pacific Ocean.