My .54 Renegade is a caplock. It took me quite a while to find a patched roundball (PRB) load it would shoot well. Mainly because I was trying to use too loose of a PRB combo, and because I didn't want to swab between shots.
I have my front sight drifted to the right. Not a lot, but it definitely isn't in the center of the barrel. My rear sight windage adjustment screw is stuck though, I probably would have the room to adjust if the darn thing would come loose. But it would be close to maxed out if it would move I think, to the left in my case.
I highly recommend ditching the store bought pre-lubed patches. They can sit around for a long time before being bought and the patches don't seem to age well siting around for a long time with lube on them. Very common issue with store bought pre-lubed patches. One container may work great, then you switch to another of the exact same thing and groups suddenly open up a lot.
My best loads. Powder charges by volume.
If 2f powder, 85 grains (it shot so well I've never tried more). It starts to shoot decent at about 75 grains, 70 grains and under were horrible.
It also shoots really well with 80 grains of 3f. 3f was also horrible if charge was 70 grains or less.
Absolute best groups are with a .530 RB and pillow ticking patching material that measures at least .017". CVA makes an un-lubed pre-cut pillow ticking patch that they advertise as being .015" thick. I consistently get .0175" thickness when compressing them with calipers. They work very well in mine but are a bit thicker than advertised.
Patches are lubed by hand with Track of the Wolf's mink oil. I use just enough to coat them entirely out to the edges on one side.
It is a tight fitting load, but looser did not give me very good groups. They take a firm smack on a short starter and then go down the bore about 8" at a time, choking up on the ram rod each time. Pushed down the bore, not smacking them down the bore.
I also have to swab at least every other shot or else my impacts take off to the left. Drastically. Some can't stand the thought of needing to swab with PRB, but my rifle has to have it for whatever reason. The jag and patch need to go down the bore smoothly but then the patch bunches and pulls the fouling OUT of the barrel, or else you may see fail-to-fire's if fouling is pushed ahead of the combo. I turned my jag down by simply chucking it into a cordless and then spinning it on a flat file, checking fit on a fouled barrel as I went. It is rare for me to have a fail-to-fire now and I don't even have to pop a cap after swabbing like many do (not an option for you I realize).
You can also try pushing a patch down onto the powder before loading the patched RB, if your patches aren't holding up this will help protect them. Many report better groups by doing this.
If your flint ignition is a bit slow that could also greatly affect groups. Just a thought, I'm not a flintlock guy.