You can't really say he's the worst champion, you can't really put all the blame on swagger though, most of the time, the contenders make the champion look good. You can put the most talented or the most "over" wrestler as the champ. But as long as he doesn't have good challengers he will not have a good title reign.
For example, in 2002/2004, Triple H and Brock Lesnar were having alot of contenders that actually made them look good, Triple H defended against alot of different big names over and over again. Like Scott Steiner, Shawn Michaels, Booker T, Kane, Goldberg, Etc.
While Lesnar he was actually having classic matches with people like Chris Benoit, Kurt Angle, Eddie Guerrero, The Undertaker, Etc. You can't give all the credit to HHH or Lesnar on those title reigns, you gotta give more cred to the contenders.
Back to topic, Swagger wasn't really helped on his title reign, all he did was defend against The Big Show and then lose to Rey Mysterio, how do you actually expect to have a good title reign like that? Swagger could have had a way better title reign if he would have had the right contenders. (Same thing happened with CM Punk and his reigns)