Becoming a good programmer is incredibly difficult and it doesn't happen quickly.
We can't expect to plant some trees and have 2000-year-old redwoods grow overnight, regardless of the demand for them.
- One basically has to be an autodidact to learn programming. It takes years of practice to learn everything necessary to get beyond just a basic level where you can write short programs that work. No one has ever become a great programmer just by taking classes or reading books. It takes hours of practice. And contrary to popular belief, CS programs do not teach programming. CS programs teach theory.
- As a programmer you need to have nearly unlimited persistence to continue trying to troubleshoot, fix, and develop things. It takes a special person to persist this much, especially when it often seems like you're not making any progress. This is pretty much a personality trait, and not having this level of persistence is enough to turn off most people who don't have it from the profession.
- You need to be exceptional at math and problem solving. Programming is a LOT of problem solving.
- You need to have an excellent short term (and long term) memory so that you can juggle multiple things in your head simultaneously, and remember what you wrote a month ago.
- You need to have a great understanding of how things interrelate and how to design good architecture. If I change this little thing here, what might I be breaking elsewhere?
- You need to have incredible attention to detail. Close doesn't cut it in programming. Forget a semicolon somewhere? Program isn't going to compile! Misspell a function name? Your program could be doing something completely different than you anticipated.
Not having these personality traits is enough to weed out most people.
- You need to have a strong understanding of data structures and classes and know when and how to use them.
- You need to have some familiarity with libraries that have already been developed so that you don't have to reinvent the wheel.
- You need to have familiarity with a lot of basic (and advanced) algorithms, again so that you don't have to reinvent the wheel.
- Often you need to know the limits of the hardware that you're working with so that you can do things like managing memory properly and avoiding running out of it, or utilizing your memory properly to eliminate wasted transfer of data within the processor and speed up processing.
- So you can program. Great! But do you know anything about packet structure, TCP/IP, HTML, CSS, user interface design, or databases? Programs don't run in isolation.
- There's a ton of stuff you need to know, and it keeps changing! It's not something you can be truly great at unless it's your main focus. You can't be a "weekend programmer."
- You need to have long blocks of uninterrupted time so that you don't lose your concentration when you're programming (and learning to program). Many work (and home) environments struggle to offer this. Phone rings? Great, it's going to take you 15 minutes to regain your train of thought.
- You often have to make decisions about tradeoffs on the fly. Sure, you could write a program that could handle every single case, but how often is someone really going to put in "zero" as an input? Besides, we're behind schedule, and it's more important to get something up and running right now.
- On top of all this, the language that you thought was the next big thing was a passing fad and nobody is using it 5 years later. Now you're on to the next cool language, which might not be used 5 years from now. You constantly have to stay on top of things.
On top of that there are the management aspects.
Programmers are often treated like factory workers. People with no programming talent (and less business sense) are often in charge of projects. They think programmers are cogs in the machines. The truth is programmers are artisans and to get the best results from a project the wise thing to do would be to ask the people who are experts at programming how things should be done! Just about every project ends up being over budget and behind schedule, forcing programmers to work tons of uncompensated and unappreciated overtime to deliver a poorly-designed and poorly-tested product. The great programmers realize that they're not being paid more than the crappy programmers because management can't tell the difference - and on top of that, they're doing extra work to compensate for other programmers being below-par. And great programmers who speak up and try to change things for the better are often intimidating to managers, who often try to get rid of them (which is a general employment trend discussed in other posts). Enough of this and it's easy to see why good people get fed up with the profession - especially since there are plenty of other opportunities. People who are even average programmers, as long as they're reasonably good at dealing with people, have enough skills to be successful in a variety of other professions. Programming is something that can be easily transferred out of, but not into.
Good programmers are probably less likely to be randomly looking for work.
It's easy to find programmers. It's hard to find good programmers. The crappy programmers are probably perpetually looking for work. The good programmers, if a company realizes that they're good, should do whatever it takes to hang on to them, so they're probably looking for work a lot less often. And when they do they're choosy about where they want to work - which would explain why many companies claim "there are no good programmers available - all we can find are crappy ones." As has been said elsewhere, a good programmer can certainly be worth several times what an average programmer is worth, and a crappy programmer can actually have negative value. And it's really tough (if not impossible) for crappy and average programmers to turn into good ones - and certainly not worth the investment for any startup to try to undertake when they're supposed to be focused on developing something quickly and getting it out the door.