Ivory tower is best used in its idiom form. The meaning is brings out the whole essence of the phrase. It simply denotes gap, maybe a large gap. Especially when it is from top to bottom.
It is a normal phenomenon in life to have hierarchy, it is the natural order of things. There is a leader to the number one man, who leads others how he sees fit. Imagine the way his subordinates look at him, like the one man that can change their life.
When he arrives, it is as if he is climbing down from the mountains. Like an ivory tower so high that his subordinates cannot climb to see him. Most of the time, he climbs down to see them.
Sometimes, the tower can be a place of refuge too. Climbing up with no one finding your safe place. It might not be literal climbing, just getting the gap you require to hide.
You can also participate by clicking here