I was in MI a handful of days ago deer hunting. Though I didn't pull the trigger I did see deer around an active logging job I was hunting, private land, relative doing the logging. I had some deer within 25 feet of me. The wind was right, but I actually didn't want them that close. At that range one could place any bullet with extreme precision, should be able to.
As a hand loader I'd look to the 60 grain partition, or similar bullet, and choose shots very carefully, no 150 yard 'hail Mary's'. I had a gunsmith that used a 243 for about everything, including routine trips to CO for elk. He swore by it, he was also very disciplined and could place a suitable bullet exactly where he wanted it. I still think that' a bit on the light side, but doable in the right hands.
Just talked to an experienced bow hunter who lost a nice 10 point just days ago. The best of plans can go awry. With the right bullet, and careful shots, the 223 would work fine. The trouble may come when that wall hanger steps out at 150 yards and he's only 30 yards from a property you are unable to retrieve from. We've all known guys that have lost deer shot at with a 30-06 too.