This has been driving me nuts for years. I really wish there were a way to disable this feature. Or if there is, if someone would let me know. I did just discover that you can press '⌘+0' to quickly get back to default size which is somewhat helpful.