Once out of a boat, you become the boat. The first step is to ensure your PFD is up to the task. Once each year, test your PFD for adequate buoyancy. When immersed in deep still water, does your head stick out of the water or have you sunk to your eyeballs. Some PFDs lose their buoyancy over time and you definitely don't want that kind of surprise when you really need to float high (like swimming a rapid on the Gauley).
If the PFD checks out, how should you swim with equipment? In general, hang on to that boat and paddle. Many paddles are black in color and really blend in well with water. I swear this is a conspiracy - LOL. Add bright stickers or some light colored duct tape in the middle of the shaft so you can easily spot a lost paddle. The boat can be an asset in big turbulent water as it provides extra flotation. Always keep the boat below you so you don't get sandwiched between the boat and a rock. When close to shore, toss the paddle on shore like a spear. You can also push the boat into shore and swim after it.
Mixing defensive and aggressive swimming techniques allows you to get yourself and equipment quickly to shore. Judicious aggressive swimming (pretending you are a boat) along with rock scrambles and wading can get you to places in a rapid that are very difficult to reach by boat. Set-up time is nil in most cases so it comes in handy when you need to get someone on a trapped boater quickly.