The castle is the core building of the inner city construction; it will provide an overview of your legion.
Upgrading it will unlock other buildings, and the current level of other buildings is also determined by the castle. Also, it will increase the size of the legion and the hospital's capacity.
It cannot be demolished.