The Imperial Castle is recorded as having had a number of different gardens during its many centuries of history, none of which have been preserved. The oldest representation of the Imperial Castle, dating from 1425, thus already shows a tree garden with a woven fence. On the south side of the great hall, Emperor Friedrich III (1440-1493) had "hanging gardens" laid out in imitation of the gardens of the Oriental queen Semiramis, which were supported on pillars and were planted with vines, flowers and small fruit trees. At the beginning of the 18th century, Johann Christoph Volkamer immortalized these gardens in his copper engraving "Nuremberg Hesperides".

Between 1538 and 1545, bastions were built on the north and west side of the castle, on which at a later date the present castle garden was laid out. On the large bastion there is a square parterre with roses, framed by trees, including pagoda trees (sophora japonica). From here there is a path leading to the southern part of the garden on the lower bastion. The circular bed in the centre is surrounded by neatly clipped maples.

In addition to these, the Imperial Castle also has the Maria Sibylla Merian Garden next to the Heathens’ Tower, which is named after the important Nuremberg artist and naturalist.

The centuries-old "Kunigunden lime tree", which had to be replaced in 1934, stood in the inner courtyard. It or its predecessor is said to have been planted by the canonized Empress Kunigunde. The castle court is thought to have convened under this tree. Albrecht Dürer also reported that there was dancing under the tree at the summer solstice in 1455. The tree planted in 1934, which had little root space above the rock, had to be replaced a few years ago.