If you look at okinawan karate (more documenation and a little more recent than many kung fu styles) you will see that originally, instructors only had 1 or 2 forms/kata. The rest was two person drills and lots of basics. The forms/katas were a system in and of itself.
If you wanted to learn more skills in an area, you would go to that instructor and learn those skills and a form/kata that emphasized those skills. Eventually instructors started to collect those forms/katas to preserve them.
Gichin Funakoshi attempted to collect many of the forms from Shuri-Te and preserve and pass them on. Mabuni collected all of the shuri-te and naha-te forms and put them in his style (around 50 kata). Some of these forms are VERY close to each other and the material is found in other places, do you really need to incorporate them? Nope, the material is already in there and the only reason is to give students more busy work to keep them happy.
I think the commercialization of the martial arts is what lead to the more is better. It gives someone tons of material to keep working on and coming back for more. Most people would rather keep learning new stuff than working on what they have to master it. Add to the fact that not everyone studies a martial art to be a fighter and you keep adding stuff for them to practice.
I think also, some forms are redundant, in that they are replaced by other forms later on in the system. I have heard of some kung fu styles that a beginner would learn a certain form, and then at the intermediate level would learn a very similiar form with some more advanced principles/concepts to it. Then finally, a third form would be learned with more additional material. So, you would have in this case 3 forms even though by the advanced level you would drop the first two in practice since the lessons are replicated.
"God gave you a brain, and it annoys Him greatly when you choose not to use it."