We don't need no stinking swim skills - we have a bombproof roll. How many of us have heard that line before? Even the best boaters come out out of their boat once in a while. There is a great podcast series called: Inbetween Swims. There are some great boaters in that series. The Upper Yough Race has the "Wheel of Misfortune" that is typically awarded to the worst beatdown during the race. I suggest taking a look at the winners of this dubious award - most are fantastic boaters. Whitewater can be very humbling - sooner or later you pay the dues for all of that fun.
Even if you don't flip, get pinned, have equipment failure, or come out of your boat - you may need to swim to implement a rescue. Swimming in fast moving water while dodging obstacles can be pretty exciting in it's own right. Endurance, planning, and knowlege are all essential in staying safe when swimming down rapids. Your PFD certainly helps and that is why we mandate that you wear it. Even with this assistance, holes, whirlpools, eddies, etc. can send you underwater when you least expect it.
When the weather gets warmer, find a nice deep stream with fast moving water and practice both aggressive and defensive swimming techniques. Learn how to ferry and catch eddies while swimming. This is a key rescue skill you should practice from time to time. Practice throw ropes skills as both the rescuer and the swimmer. Many a wise paddler has stated they will not run a rapid that they can't swim.