Is there anything more mom-and-apple-pie in American business than obeisance to the great god, Innovation? Isn’t innovation what it’s all about? Can’t innovation itself be a corporate grand strategy. On all sides we hear: “Be first to market.” “We out-innovate our competitors.” Hewlett-Packard’s slogan is “Invent.”

Wouldn’t it be crazy to try to convince anyone that this god was a false idol?

Yet I recently completed an assignment that left me convinced otherwise. No doubt innovation is astoundingly important for many businesses to remain competitive and have a chance at respectable—or outrageous—profitability. But it can just as easily be strategic quicksand for many companies.

The first alarm bells went off when I learned of a study by Boston Consulting Group that showed convincingly that dollar-for-dollar, large pharmaceutical companies with R&D departments funded in the hundreds of millions of dollars are no more productive at getting new drugs to market than much smaller companies. That means these companies aren’t realizing the expected benefits of scale in their R&D efforts. All those expensive machines, those massive biological and chemical libraries, that expensive research talent has been no more productive per dollar spent than smaller companies targeting much narrower market niches.

Even more astonishing is a realization based on the fact that pharmaceutical business is more like George W. Bush’s old career of oil wild-catting than one might think. Very few—some say 1 in 10,000 investigated chemical compounds—become a successfully marketed drug. But this means that a company with a large R&D budget should be able to diversify its search for a successful, let us say, anticancer compound—while the little company must take a higher risk that its narrower bet on a pharmaceutical approach will pay off. But this is not what we’re seeing.

What we are seeing is large pharma companies buying the rights to market or co-develop drugs with much smaller companies. Or just outright buying them.

This means that innovation can’t be the pharmaceutical industry’s strategic golden key.

The second alarm bell rang during my investigation of innovation in a related field, medical devices. A senior R&D executive with this well-known multinational company, Company A, had typical innovation problems. New products were late to market. Superhuman efforts were required to corral the marketing, engineering, scientific and manufacturing talent in one place to work on a product concept and breathe life into a prototype. If the product was on time, it was over budget; if on budget, then late. Or, not infrequently, both.

Compare this with the medical device division of Company B. Company B insisted that it had “no or very little in the way of new product development problem. This household name company belched forth-new products on budget and on schedule. That was a surprise. Few companies can say that—since everyone recognizes that innovation is, or at least should be, an inherently risky business.

All became disappointingly clear when even its new products czar admitted that senior executives required a business case so tight, and an engineering plan so perfect, and a manufacturing sign-off so firm for new products that it lost a crucial 8-16 months getting new products to market. In other words, the company placed only “sure bets.” In fact the innovation czar typically spent 4-6 weeks solely on the “horse-trading” it took to borrow the right people from their ordinary jobs to serve on the new product development team. All in all, the company had trades off timeliness and true innovation to avoid failure.

Yet in the end, this innovation strategy doesn’t always lead to a perfect product launch. Or even a safe one. The company was recently fined $60 million for fielding a medical measuring device that delivered misleading readings—and covering up the fact from the Food and Drug Administration. It may face multimillion dollars in consumer lawsuits as well.

Was this all due to simple mismanagement of the product development process in these companies? Should we just expect great executives to do better? Or was there something deeper at work? Something that cuts to the black heart of innovation-as-strategy?

Odysseus went to Hades to find out the value heroism after life on earth . I went to Silicon Valley to find out the truth about innovation. I found the following insights in a conversation with Mr. Dan Cook, Nortel Network’s brilliant strategist.

The first thing Cook clarified was the hidden reason why so many companies spend so much on R&D and get so little for it. It’s not just that the talent needed for product development is hard to “blast out” of their functional area assignments to spend unclearly-defined and sometimes un-rewarded time on new product development teams. It’s that much of the time spent in R&D is focused on three things:

• “R&D” on existing products for existing customers: fixing stuff that’s been in the field for months, years, decades but still has problems or needs to be upgraded to meet current customer requirements

• “R&D” on products that were launched crammed with the latest compromises among R&D, marketing and manufacturing and don’t really work right

• “R&D” on making a new generation of products backward compatible with existing products—so as not to discomfort an existing customer base.

And indeed when I talked to a firm whose expertise is in speeding up new product development , I confirmed that often 70-80% of the time spent by R&D departments is spent on these three backward-looking efforts. In fact, this firm won’t take an assignment from a client unless top management is willing to put this proportion of the R&D budget on the operating table.

The second thing Cook pointed out was something I’d heard before, and believed—but he raised the temperature on. And that the fact that most companies who rely on innovation do too much of it.

What does this mean? It may be Christian teaching that many are called but few are chosen. But in most R&D intensive companies just the opposite is the case. Too many R&D projects linger long past the time there is much hope they will bear fruit. They aren’t killed off soon enough. Why?

• Nobody likes to be the bearer of bad news, especially after executives and researchers have spent a good portion of their careers, their reputation and the company’s money on a dream
• Every project by its nature builds a constituency. After all jobs and careers are on the line
• It’s human nature to avoid writing off a sunk cost. “We’ve put $30 million into this—It’s gotta be worth another 5. We’re so close to the X milestone!”
• The question of when to kill a project is inherently ambiguous. It’s easier just to extend it a little longer. The Pentagon isn’t the only organization fighting this infection. Just the most well-known.

The result is that the R&D budget gets divided into finer and finer pieces. So R&D as a whole becomes less and less productive. After a certain point, not only is there not enough money to fund a new project decently—there’s not enough money to even pursue the remaining glimmering promise of an old project decently. So neither new nor old products win.

These two points of Cook’s explain—at least in part—the weak productivity of larger R&D-based firms, like those in our pharmaceutical example. Just because they are large, they have enough resources to spread around on old projects that should have been killed long ago. Smaller ones don’t. In fact in smaller pharmaceutical companies, if the R&D project doesn’t work out, its not just the project that gets killed. Its can be the entire company.

So, all-in-all contrary to conventional wisdom, it’s far too simple to say that “we need more innovation.” We need more productive innovation. We need to kill more projects. We shouldn’t say, with Company B, that “Our kill rate is only 10%.”And with considerable heresy, we need a way to turn our backs on our existing customers.

In Business Maverick #9 we’ll talk about some real-world answers to both of these innovation contradictions.