There are many options available to keep your core warm. When starting out you can certainly save money by purchasing wet suit instead of a dry suit. Wet suits are very easy to repair and provide excellent abrasion resistance. Wetsuits also require very little maintenance. A decent wetsuit will last decades. Most rescue squads have chosen the wetsuit for this reason. The Farmer John style provides excellent warmth and doesn't restrict mobility. Don't choose a scuba diving wetsuit as they are typically too thick, stick with a 3 mm thickness. Wetsuits cost $100 - $150 new and much less used. Dry suits are the ultimate in comfort, except perhaps that choking feeling around the neck gasket. Latex gaskets surround the neck, wrists, and ankles (better yet, get integrated booties) keep water out. Some have breathable material like Gore-Tex to let water vapor escape. Dry suits require a separate insulating layer, they are just a water barrier. Dry suits require maintenance, are challenging to repair, and don't last as long as a decent whitewater wetsuit. Most paddlers these days go with dry suits as they are far more comfortable and warmer. Dry suits cost $500 - $1,000.
Polypro (or wool) have been used for many years as insulating layers. This type of material lasts many years and is quite inexpensive. Typical cost is $10 - $20 per garment. Cheap wool sweaters used to be more common in years past. Specialty synthetics have mimicked many of the wet insulation properties of wool and are far less bulky. Synthetics can be tossed in the washing machine and don't shrink like wool. One material to avoid is cotton (except perhaps in the summer). Cotton provides no insulation when wet and in some cases it can shrink when frozen.
Fuzzy Rubber, Hydroskin, Mystery Fabric are various high tech insulating layers. All work great and can be used for three season paddling. Prices range from $50 - $150 and typically last many years. Northwest River Supplies is a great place to check out your options. REI, Hudson Trail Outfitters, and Potomac Paddlesports are excellent places to shop as well.
Rash guards like the shirts from Immersion Research or Under Armor are excellent for sun protection and will not overheat you. They typically run about $30.
Splash tops block the wind. You still need an insulating layer underneath to provide warmth. This can be as simple as a cheap nylon windbreaker from Wal-Mart or specially designed for paddling. Ones made for paddling typically come with neoprene wrists and collar to keep some of the water out. Paddling jackets cost $50 - $100 new.
Dry Tops cover only from the waist up. Just like dry suits, you still need an insulating layer underneath. The wrists and neck are latex gaskets to prevent any water seepage. They also have a tunnel system which allows the jacket to sandwich your spray skirt tunnel. These jackets are incredibly warm and many use for four season paddling. All is fine unless you swim or need to help out in a rescue. If you go this route, consider wearing long pants made with a neoprene material or better yet a full farmer john style wet suit. Prices new are typically $200 - $400.
The best advice I can give when dressing to paddle in colder weather is dress in layers and dress to rescue as in full immersion. Here is a good video on how to dress for rescues: Dressing for Rescues.
