Margrave Georg Wilhelm founded the Hermitage on the site of a zoo belonging to his father Christian Ernst. In 1715 the Old Palace was begun as the focal point of the hermitage. In the palace and the scattered hermit's huts in the woods the court played at being members of a hermits' order.

From 1735 Margravine Wilhelmine added further formal sections with boskets, avenues and water features to the gardens originally laid out by Margrave Georg Wilhelm. Although traditional baroque elements were used, the result was not a typical baroque garden: the complex is not dominated by a main axis and the individual garden sections are more independent from one another than was usual in the high baroque era.

The New Palace with the upper grotto, the lower grotto, the Margrave's Hermitage, the "Ruined Theatre" and other small buildings from Wilhelmine's day are still the dominant features of the Hermitage. At the end of the 18th century, the Hermitage Court Garden was transformed into a landscape garden, which included replacing the boskets with natural stands of trees and meadows. Over the last 30 years, individual sections of the park which disappeared in the 19th century have been reconstructed.