Today I had an excellent, wide-ranging discussion with Forrester analyst Bill Martorelli about outcome-based outsourcing.  Bill is currently working on an upcoming research report about outcomes which he will be presenting at the Forrester Services and Sourcing Forum 2009 and we were comparing notes and I was looking for some feedback on the messaging I’ve developed for my company, Symphony Services.  We had a pretty big disparity in our estimates.  We were talking pretty broadly about the ITO and outsourced product development (OPD) markets, but I said that it’s probably less than 5% of total contracts, definitely less than 10 percent.  Bill thought that it was larger, perhaps twice that, but as we talked about what really counts as true outcome orientation versus simple output-based, fixed-time/fixed-price or SLA-supported contracts, he felt that my number probably wasn’t too bad.

But what this really highlights is that even some of the brightest minds in the business don’t have a good handle on how much of the outsourcing contracts are outcome-based.  What do you think is the right number, or do you think it’s just so much hype?

Tying the costs of outsourcing to the achievement of outcomes that support real business objectives sounds like nirvana.  You can assign a value to a given activity and it can help you better evaluate the ROI that you’re getting too.

Not surprisingly in this economy, outsourcers are trying every angle they can to get business and many are tired of just reducing rates.  So dont’ be surprised if they start making promises about delivering against outcomes and outputs.  But don’t just believe the hype.

The real question that you have to get to is how committed are they to really delivering on outcomes and how much are they just trying to suck you into a sales conversation, only and perhaps purposefully, to shift the conversation back to traditional outsourcing engagement models.

Do your own due diligence and understand how committed they truly are to this kind of engagement.  Here are a few questions to ask

• What they’ve changed organizationally to enable them to deliver against outcomes instead of providing bodies?
• What % of their business is coming from outcome or other performance-based arrangements
• Especially for vendors that have a heavy offshore component, how are they dealing with the philisophical and cultural shifts required to really deliver.

What you may find is that their’s not much beneath the veneer.  In the meetings that our CEO Gordon Brooks and I have had with journalists and analysts, we’ve of course gotten very good feedback on our approach to outcome certainty (especially in BusinessWeek’s NEXT blog).

But what’s really struck people, like AMR’s Dana Stiffler, is the extent that we’ve embraced this approach.  Today about 20% of Symphony’s engagements are outcome or output-based with another 40% of so utilizing other performance-based mechanisms like revenue sharing, fixed margin and SLA’s.  According to Stiffler, she hasn’t heard of anyone else in the software product development outsourcing space really embrace outcome-orientation at all and even in the big Indian IT shops its a tiny percentage of their business.

Now why don’t other firms adopt performance-based contracts as aggressively — because it’s hard.  It’s hard to track the metrics that matter.  It’s hard to change the way that your employees think about thier work to align with delivering outcomes.  And most of all it’s hard to change the risk profile that your company is used to when taking on these committments.

But there are a couple of key points that I think Dani misses on.

First, the mastodon in the room to borrow a phrase, is that most ISVs have an overwhelming desire to protect  current revenue streams (i.e. the status quo).  They don’t have the real committment to or belief in SaaS, but offer it as an afterthought, if at all.  And often it’s done under the auspices of opening new market segments (e.g. SAP’s contention that their Business ByDesign SaaS offering is an alternative for small to medium businesses that can’t afford their core product) or as a defensive measure.  With that perspective the organizational changes that Dani talks about will never be achieved.  But that’s OK because the wholesale move to SaaS is not the desired outcome.

In Dani’s example of a company that made the committment to shift to an On Demand model, he highlights the fact that the Board brought in a new CEO and changed almost the entire executive staff — save the VP, Engineering.  I don’t doubt that these moves were criticial to success in that situation.  Now I understand that perhaps there’s a feeling that you need to keep the technical talent that knows the product.  But this is a decision that I think can really kill the transition from OP to OD.  Dani was 100% right about need to to have a “service” view versus a “product” view and understanding the need for “serviceability”.  But this leads me to my second point.

As I had written before, one of the critical philosophical  changes that engineering organizations must make is to move from a purely software engineering mindset to a systems engineering mindset.  It’s so important that the engineering teams architect and design an On Demand system with data and application availability, reliability, security in mind and understand how changes in the application can affect operations.  If the engineering team gets that wrong all the other organizational changes will be for naught.

Time and again as we work with ISVs to evaluate On Premise to On Demand product transition strategies, we see that the evaluation of the current product’s ability to support an On Demand model is under-estimated, resulting in unrealistic expectations.  Therefore, making sure that the engineering organization has the underlying On Demand DNA is extremely critical to a smooth transition.

In the world of software testing, a strong and well executed test automation program can act as a force multiplier for not only your QA organization, but the entire R&D team.  With an aggressive test automation program you can:

• Reduce test cycles by at least 40%
• Slash testing budgets by removing the time and effort from manual testing
• Vastly expand code coverage

But where automation becomes a real force mutlipler is in catching more bugs before the code is released into production!  Industry data shows that the cost of remediation can be 100x higher post-release than at the beginning of the product development cycle.  By using automation to find the bugs earlier in the cycle you can greatly reduce re-work (not a value-added activity in the first place) and sustainting engineering costs.  On top of that you can cut support volumes and costs by up to 75% (Gartner), not to mention saving yourself the damage to your brand and CSAT by having bad product in the field.

Even just on a pure QA testing budget perspective, Symphony Services is driving ROI’s of 2-3 months, saving in one case about $100K/month in testing costs with our test automation services.

So it all sounds great, right?  Yet when we engage with clients and prospects we find that while some have some level of automation, it’s often quite low, with less than 25% of all test cases automated.  And of course many others haven’t implemented automation at all.  Why is this?  I’ve got a couple of ideas:

1. Lack of Automation Expertise: This is of course the most obvious reason why people don’t use it aggressively.  But that shouldn’t mean that you should eschew test automation as a strategy.  There are plenty of companies like Symphony Services who have the right expertise, and automation frameworks to help — although not as talented
2. Wrong Automation Strategy: This of course pre-supposed that there’s a strategy in the first place and not a decision to just start automating cases.  This problem manifests itself through Wrong selection of test cases, inadequate maintenance considerations, insufficient ROI models lead to inefficient test automation with low ROI yield.
3. High Upfront Automation Costs: That is if you’re starting from scratch.  Highly maintainable scripts require careful planning, development of reusable framework & components.  If you don’t have these handy, this increases upfront investments and reduces time to value.  Additionally, you need to drive towards lights-out execution to really ratchet up the benefits, but the ability to do that needs to be designed in at the beginning.
4. No Attention to Script Maintenance: This is what usually kills an automation project.  Teams begin an automation project, but then can’t keep up with the script maintenance.  So the scripts get out of date, the benefits begin to melt away and the project seems like a failure and is abandoned.  Common approaches such as Record & Replay yeild fragile automation scripts and don’t work.  You need to access re-usable libraries and modular, self-sufficient scripts.

So what’s holding you back from implementing test automation?  Of course if I’m all wet, let me know that too.

Please write back and let me know your experiences with test automation.

1. I agree that many large companies struggle with developing innovations and that most companies focus on feature/function improvements that are just a function of corporate inertia.  But I also think that this discounts a very important problem: many software companies don’t have a good innovation management process. This is the probably the single biggest reason for failure to deliver impactful innovations and the most underrated aspects from an organizational perspective. If you don’t have a working funnel to evaluate new ideas and winnow them down to the few that get prototyped and implemented, you are left with a situation where you are hoping that the best ideas (or any ideas of merit) actually make it to your customers.  Worse yet, you lose the ability to track value of ideas generated and implemented, calculate an iROI (Innovation Return on Investment) are build institutional knowledge of what works and what doesn’t to inform future investment decisions. BTW, at Symphony Services we have another way of evaluating the return on innovations from a revenue perspective called the Vitality Index, as opposed to looking at cost driven evaluations like R&D  as a % of sales, # of patents, etc..
2. I also think we need to explicitly recognize that while start ups are usually the source of many of the discontinous innovations, many start ups are founded by employees who worked at bigger companies, and came up with good ideas that were not acted upon by the big companies in which they worked. So they leave, create their own company and try to bring them to market.  Even some of the most progressive companies like Google, who promote the fact that they want their employees to spend 20% of their work time on coming up with new ideas, leave thousands of “inventions” (it’s not an innovation unless it generates revenue or reduces expenses) on the table each year (sometimes for good reason) and people still leave to develop their own ideas.  This is actually a very beneficial thing in my view and keep a healthy innovation ecosystem alive and well.
3. Michael also points out that due to the premise of delivering better, more predictable returns for their shareholders, the big mega-vendors don’t make the investments in R&D to generate these new innovations.  I don’t think that’s actually true.  Microsoft spends billions, but seems to get very little in return.  Many times I guess they’re just spending on the wrong things.   For the large companies, I think it’s generally cheaper — and easier — to let others invest in new ideas and then just acquire the companies that are successful.

TechCrunch’s Sarah Lacey talked about a coming IPO boom, noting that IPO registrations are up, but it isn’t only for the uber-platforms like social networking giants Facebook and LinkedIN, but a broader rally that includes smaller companies like OpenTable.

In talking to clients and prospects we’re also seeing a thaw in the market and companies are gaining confidence and visibility into the future, bringing projects back on line that have been on hold for a while.  Are you seeing the same positive signs at your company?