Khandar fort was built by Kshatriya Badgujar rulers .But eventually due to internal conflicts between Rajput rulers Badgujar lost its control and thus was a major setback for Badgujars. After losing there lands in fight against kushawaha rajputs Badgujars migrated to neighbouring states of Madhya Pradesh, Gujarat , Maharashtra . Located on top of a strategic vertical hill, the Khandar Fort in Sawai Madhopur could never come under an easy attack and thus was truly regarded as invincible and was a favorite of many dynasties in India. It is also said that king of this fort never lost in the war. This fort also tells some historical stories by the damaged walls and gates.
The fort was long ruled by the Sisodia kings of Mewar after which it was taken over by the Mughals for a short while before coming under the direct control of the Maharajas of Jaipur in the 18th century. This fort also belonged to Badgujar Kings and was attacked by Allaudin Khilji.