Khan was born in Kasur, then in the Punjab province in British India; after partition, Kasur became a part of Pakistan.[2] His father was a famous singer, Ali Baksh Khan, in a West Punjabi family of musical heritage.

At the age of seven, he learned sarangi and vocal from his paternal uncle Kale Khan, who was also a singer and composer of repute, and after his death he picked up a few compositions from his father who had re-married by this time.[3] This tenure lasted for almost three years. It was at this juncture that he improvised the musical instrument "Kanoon" to the present "Swaramandal" which became his trademark. When he was about 21 years old, he moved to Benaras where he accompanied a courtesan named Zarabai on the sarangi and started to perform in public.
Singing career

Though Khan started his career by accompanying female singers on the sarangi, he also used to sing a few compositions of his late uncle. He was also a disciple of both Ustad Akhtar Hussain Khan and Ustad Ashiq Ali Khan, two scions of the Patiala Gharana.[4] He became popular as a vocalist after his debut concert in Kolkata. Khan amalgamated the best of four traditions: his own Patiala-Kasur style, the Behram Khani elements of Dhrupad, the gyrations of Jaipur, and the behlavas (embellishments) of Gwalior. His voice had a wide range, spanning three octaves, effortless rendition, sweetness, flexibility and ease of movement in all tempi.[3] His raga expositions were brief, contrary to convention, and while he agreed that the beauty of classical music lay in leisurely improvisation, he believed that the audience would not appreciate long alaps, and he had to sing for the masses and change the music to what the audience wanted.

After the partition of India in 1947, Khan went to his home in Pakistan, but returned to India later to reside permanently. He did not approve of the partition, stating: "If in every home one child was taught Hindustani classical music this country would never have been partitioned." In 1957, with the help of Bombay chief minister Morarji Desai, he acquired Indian citizenship and moved to a government-provided bungalow at Malabar Hill. He lived at various times in Lahore, Bombay, Calcutta, and Hyderabad.

For a long time, Bade Ghulam Ali stayed away from singing in films despite requests and persuasions from well known producers and music directors. However, he was coaxed and convinced by K. Asif to sing two songs based on ragas Sohni and Rageshree for the 1960 film Mughal-e-Azam, with the music directed by Naushad. He demanded a very, very high price, reportedly Rs. 25,000 per song, when the rate for popular and star playback singers such as Lata Mangeshkar and Mohammed Rafi was below Rs. 500 per song.

Khan was awarded the Sangeet Natak Akademi Award and the Padma Bhushan in 1962.

Khan died in Hyderabad at Basheerbagh Palace in 1968, after a prolonged illness which left him partially paralysed in his last years. He continued to sing and perform in public with the support of his son Munawar Ali Khan (1930–1989) until his death.