I was faced with this decision about 25 years ago when just getting started into ML and looking to buy my first rifle. I wanted to use it with conicals for elk in Colorado. The prevailing wisdom at the time said that the .54 was better for elk, so I went with that in a TC New Englander. Later I learned that, yes, their is some benefit to the .54, but it is quite slight. About that same time, the movement toward inlines hit, most of them in .50, so I switched to that, and of course they work fine for elk.
I applaud your decision to simplify your life and move toward just one caliber. When you're dealing with various molds, alloys, lubes, sizing, wads, etc, etc, having two calibers exponentially complicates matters. Trying to organize everything becomes a nightmare - it's hard enough with just one caliber!
A lot of it depends on your temperament. Some people don't mind the extra bother!