Having written the heading I suddenly wondered where the term “ivory towers” comes from. Wikipedia says it has a Biblical origin (from the Song of Solomon) but,
From the 19th century it has been used to designate a world or atmosphere where intellectuals engage in pursuits that are disconnected from the practical concerns of everyday life. As such, it usually carries pejorative connotations of a willful disconnect from the everyday world; esoteric, over-specialized, or even useless research; and academic elitism, if not outright condescension. In American English usage it is also used as shorthand for academia or the university, particularly departments of the humanities.