Ustad Rajab Ali Khan was a son and disciple of of Ustad Mugalu Khan of Kirana tradition. Later he got training under Bade Mohammad Khan of Gwalior Gharana for vocal and Amritsen of Senia gharana for Been. He was also trained by legendary Beenkaar Ustad Bande Ali Khan Sahab. He was a master of very vibrant, complex & speedy Taan. He was also well-versed in many rare Ragas & Bandishes. Besides singing, KhanSaheb used play various instruments such as Been, Sitar, Jal Tarang, Tabla very efficiently.

Rajab Ali was a court singer at Devas state. For some years, he was also court musician at Kolhapur, where he developed a clashing interaction with Ustad Alladiya Khan. He spent most of his life back in Devas.

His major disciples include Amanat Ali Khan, Ganpatrao Devaskar, Nivruttibua Sarnaik, Rajabhau Deo, Krishnarao alias Mamasaheb Mujumdar (father of Kalpana Zokarkar). Ustad Amir Khan of Indore and Ustad Salamat Ali Khan of Shamchorasi Gharana were deeply influenced by Ustad Rajab Ali Khan's Taan. Pandit Kumar Gandharva was also influenced with Rajab Ali Khan's music.

He died in Devas at the age of 83.