“Most of the time [companies] think they can do one or two services, and then they’ve got SOA,” says Frank Cohen, founder of PushToTest, which specializes in the testing of SOA, Ajax, and web services. “We also find that typically there’s no blueprint to explain how it all hangs together.”

If companies fail to realize the operational nature of SOA from the start -- the interconnection of multiple services, the interface changes that happen over time, the need to be able to switch to an alternate service if the main one is down -- “they bump into very sleepless nights and angry customers because an application isn’t available,” Cohen says.

Blame many of the problems on a lack of standards, he says. “Without a standards body you can make assumptions about what SOA means and get the implementation wrong. Without a standards body, there is no generally agreed upon SOA reference application that people could benchmark against and produce best practices and optimizations around.”

Enter PushToTest’s SOA Knowledge Kit, the company’s first SOA use case that presents its findings around implementing products from BEA, IBM, Oracle, and Tibco in the same scenario. It has published all of the code, methodology, test suites, and deployment scripts during this project under a free, open-source license, so that organizations can build and operate the use case implementations in their own environments. Full disclosure: Tibco commissioned the research, but Cohen says that PushToTest makes no effort to improve the results for the commissioning company, which has the right to release them or not. It found all the vendors provided competent solutions, though Tibco ActiveMatrix took less time and cost less.

Businesses can get more details on the results and access to the kit here, and Cohen says he invites the other vendors to use their own tools and come up with a better implementation to measure against.

Generally speaking, Cohen sees some spots where vendors need to pick up the pace. Consider the issue of how many services users typically have under management in a registry.

“When I ask about this, the typical answer is 1 or 2. Occasionally someone will say we have 1,200,” he says. “Very few people understand the real benefits of a registry.”

To that point, he says that vendors have failed to capture the imagination of developers, testers, and architects. “The mark that registries have taken off is when the average Java developer sits down and says, where is the registry to see which components have already been built? That doesn’t exist yet,” he says.

The good news, he says, “is that we were able to accomplish the use case on all four platforms. So it is possible to build SOA today.” But the challenges for users aren’t trivial. “When you look at key drivers, most of the CIOs used to live behind corporate firewalls with simple development needs,” but today they are flooded with internal requests for more sophisticated services that mirror what users are used to on the public Internet, like YouTube and Flickr.