How comfortable are you upside down in your kayak? In my beginners class, I have the students move to the tucked position and tap the side of their boat 10 times before performing the wet release exercise. It gives them something to do while upside down and helps them overcome the fear of being underwater. Sometimes it makes sense to take your time in set-up and wait a bit for that nice solid water feel before executing your roll. If being upside down frightens you, you will most likely rush your rolls or pull the rip cord prematurely. How do you gain that confidence? Practice rolling in waves, practice backferries in fast moving water - always a great carnage exercise. Some other exercises for gaining water comfort are hand paddling in easier rapids, catching eddies with just your hips, surfing waves and small hydraulics.
Confidence is like constructing a building - you need a solid foundation. Your foundation consists of training, practice, and variety. Take turns as the lead paddler or choose different lines rather than following others. Take your time stepping up river difficulty, there are lots of great runs for every skill level. Ignore others progress, many rise too quickly only to crash and burn - this sport is supposed to be fun. Here is a great set of articles on confidence: How to Gain Confidence.