In 1066, William the Conqueror came to Dover after the Battle of Hastings to capture the port. He established a fortification, possibly around the church, but there are no surviving remains. The castle was extended in the 12th century, although we know nothing of its appearance before the great rebuilding of the 1180s.

Immediately following his victory at the Battle of Hastings in October 1066, William the Conqueror marched on Dover Castle, recognizing its supreme strategic importance as the "Key to England." Upon his arrival, the English garrison was reportedly so stricken with fear that they prepared to surrender unconditionally. However, according to the chronicler William of Poitiers, Norman soldiers "greedy for booty" accidentally set fire to the castle, causing significant damage. Recognizing that Dover was the shortest crossing point to mainland Europe, William did not abandon it; instead, he spent eight days personally overseeing its fortification and repair. He likely built a "motte and bailey" style structure, potentially utilizing the existing Saxon earthworks and the Roman pharos (lighthouse) already on the site. His actions at Dover secured his supply lines to Normandy and served as a powerful defensive hub that allowed him to confidently advance toward Canterbury and London for his eventual coronation on Christmas Day 1066.