The manor, known as Mūriniai Videniškiai, was fortified by 1585 with the work of Swedish masters, funded by Vilnius castellan Matas Giedraitis and his son, Martynas Marcelis Giedraitis, Voivode of Mstsislaw. It was part of the defence system of central Lithuania and protected the old Vilnius–Riga route. The castle consisted of two two-story masonry buildings surrounded by ramparts, with gates and underground storage rooms within the fortifications, as well as wooden residential and service buildings. At the beginning of the 17th century, manor palaces and a Renaissance garden were constructed to the north of the castle, on a plot encircled by the Siesartis River. From the mid-17th century, the castle lost its defensive function.
