Elesa was really annoying...but she never made my pokemon faint. Clay was annoying because of the ground move Bulldozer. I agree, they should make the next one harder to beat.For me, all of the Gyms were insanely easy with the exception of Elesa and Clay. Elesa is only difficult without a Rock move to take out those pesky Emolga, and I only found Clay hard because of his Excadrill being a Steel type. Still, Herdier's dig did the job on Excadrill. I was really surprised.
Skyla and Drayden/Iris were particularly easy. On my White game neither of them launched a hit on me. =P
I really hope the inevitable third version boasts a much more fearsome Gym roster. I was disappointed in how easy the gym leaders were.