This is something I'm going to start doing too, just to see how much difference it makes, I've never bothered to do so.
I'd say play around with it. Sort them to within 1 full grain of each other and see what groups look like. Then maybe try a 1/2 grain spread and see how much it made a difference.
Being as consistent as possible lubing patches helped me a lot with flyers. I will say up front that I got the most flyers from store bought prelubed patches. The Dutch Schultz method is the most consistent way I know of to lube patches but if you're careful other lube methods can be good too. I can vary my group sizes by 2"-3" just by the amount of lube I use. At least with my factory 1:48 TC with shallow grooves anyway. Too slick seems to make it "skip" through the rifling, groups are definitely bigger if I use quite a bit of lube. Using mink oil from TOTW, if I put on enough lube to "fill the weave" of the patch like is often recommended groups are a couple inches bigger than if I just barely lube the patch. By barely I mean I do one single swipe across the lube with a patch, using the same pressure every patch, and then attempt to rub what little there is out to the edges. You can barely tell they have lube on them but shot patches look good and groups definitely tightened up a bit for me. Using the Dutch Schultz method I like a 6:1 ratio. If you're unfamiliar with this, you mix a ratio of Ballistol and water together (or Castor oil and denatured alcohol can be used). The bigger ratio is the Ballistol or Castor oil. Shake the heck out of it to mix them together well, let the foam settle a bit (Castor oil / Denatured alcohol don't foam nearly as much as Ballistol/Water) and then dip a patch into the mixture one at a time and then set them on a screen or wax paper. You let them sit out until all of the water or denatured alcohol evaporates away leaving only the Ballistol or castor oil on the patches. Good way to get the same amount of lube on every patch. Known as a "dry patch" system. Generally folks find that a ratio somewhere between 5:1 and 8:1 is best. Two things to be aware of: You'll need to swab between shots with this method or fouling will get terrible, and don't make up a bunch of patches way ahead of time because it will deteriorate the patch material and groups will really open up if the patch isn't holding up. I don't like to make up patches more than a couple weeks in advance of using them.
I find swabbing every shot also far and away yields better groups for me. I get some crazy flyers if I don't swab. I swab the same every time, one pass down and back with a barely damp patch (pausing at the bottom for about 6-7 seconds to let the fouling soften a bit) and then a pass down and back with a dry patch. Swabbing every shot can result in pushing fouling down and blocking the flame channel, so either popping a cap after swabbing to ensure it is clear or turning your jag down a bit so the patch and jag go down smoothly and then the patch bunches up and pulls the fouling OUT of the barrel is best. I chucked my jag into a cordless drill and spun it on a flat file to turn it down, going a little at a time and then checking fit with a cleaning patch on a fouled bore. I tapered it a bit, thinner in the front tapering to a bit bigger in the back.
Weighing out charges also gives better consistency than using a powder measure. You can use a measure, being as consistent as possible with filling it, and find what charge shoots best. Then fill the measure with that charge, dump the powder out onto a scale and weigh it so you know what weight you're looking for to measure out all your charges exactly the same. IME a grain or so of powder difference doesn't really seem to make a lot of difference in my groups with black powder though, but being consistent is the name of the game when looking for tiny groups on paper.
