I don't see any downsides to swabbing either, provided that the patches are moist rather than sopping wet, and provided that dry patches are used to remove as much moisture as possible to avoid powder neutralization.
It seems that most people who find success by putting wads under full-bore conicals are using FLAT BASE conicals, such as the Lyman Great Plains bullets that you are using.
I'm using Hornady Great Plains bullets, which have concave bases. I posted what I found when comparing accuracy of HGPs with and without wads to the thread "Wads and PRB / Great Plains rounds". Here's an excerpt from that post:
"Using a wad between bullet and powder was suggested by several knowledgeable members of this forum, so yesterday I tried it out. With an HGP sized to .501 and pan-lubed with SPG-Idaho (8 oz SPG, 1 oz ALOX, 2.6 oz Stihl Ultra HD 2 cycle oil), over 70 grs W (measured by weight) of T7-2F, I was able to shoot a group measuring 1 1/2 inches at 50 yards. Measured velocities were between 1379 and 1432 FPS. Keeping everything the same, but adding a 1/8 thick .54 wool wad between bullet and powder, with the wad loaded as IdahoLewis suggested in an earlier thread, the group size for my next 5 shots blew up to 7.0 inches (!), with three of the shots on the margins of the group. Measured velocity range for this group was 1389 - 1425."
This is very similar to that I found when putting wads under unsized HGPs.
The issue, I'm guessing, is that with a concave base, it seems very unlikely that the wad and the concave base of the bullet will be exactly aligned, and so one edge of the wad will engage the rim of the concave base less heavily than the other edges. Pressure from the burning powder will therefore push the more lightly engaged edge of the wad into the concave base first, resulting in uneven pressure on the base of the bullet, uneven expansion of the concave base to engage the rifling, and a bullet which is out of round as it exits the muzzle of the rifle.