Is there anyway to turn that off? To turn it back to just scaling the text rather than the whole document?

Why is zooming the whole document the preferred method? It feels sub-optimal to me, but maybe I'm missing something.