The harshness got bad when I installed the swaybars, UCA and lca.
Hmm. Well, you've already removed the LCA (I assume you had installed - and then removed - the relocation brackets at the same time as the LCAs) and the poly diff bushing. So that leaves the UCA itself and the mount and the swaybars.
You mentioned the endlinks on the rear bar but you didn't mention the front seat bar links. If they are stock, maybe it's possible that you have too much preload on one or both sides of your bar that is contributing to the harshness?
Can you tell if the harshness when you hit a bump is mainly from your front suspension or rear?