The earliest settlements in Vilnius are believed to have been found in the valleys of the Neris and Vilnia. Archaeological findings near the Old Arsenal trace back to the 4^th century BC and findings in the Castle Hill (Lithuanian: Pilies kalnas) are dated the 1^st century BC. In the 5^th century AD, there was quite a big settlement at the foot of the hill. In the 15^th century, there was the Upper Castle built from bricks on the Castle Hill (now often referred to as the Gediminas’ Hill); the wooden Crooked Castle (Lithuanian: Kreivoji pilis), called as such due to its shape, was built on the nearby Hill (Lithuanian: Pilkasis kalnas), where the monument of Three Crosses (Lithuanian: Trys kryžiai) is situated now.

The only tower of the Upper Castle, which has survived until nowadays, is Gediminas Tower. The Upper Castle has been referred to in written sources from 1323. After a big fire in 1419, the castle was rebuilt by Grand Duke of Lithuania Vytautas, who also built a luxurious palace of grand dukes in the territory of the defense castle. The castle was ruined during the war with Moscow