In a previous post I wrote about IBM buying Lombardi, and noted that I expect to see more industry consolidation.  Well, we didn’t have to wait long to find out which merger is next.  They say things  occur in threes, so I wonder which BPM company is next….

The IBM acquisition of Lombardi was interesting because IBM needed to fill the department level workflow product need.  This is where Lombardi is being positioned, according to IBM.

Progress software’s move for Savvion was much more interesting because Progress did not have a BPM system, or history of workflow systems.  Instead, Progress has a wide range of SOA systems and tools, and a strong offering in the CEP (complex event processing) and BTA (business transaction assurance) space.   With these offerings, Progress would more likely be positioned to sell products and services to the IT architecture crowd more so than the business.  Now with a full  stack of BPM, CEP, transaction monitoring, ESB, and data services, Progress is very well positioned to be an aggressive competitor in the BPM market.

I’m very interested in this Progress and Savvion merger because it shows the BPM industry is evolving into more event oriented processes instead of people centric activity.   CEP (complex event processing) refers to the idea that things happen that are not necessarily all part of a process.  For example, I can correlate streaming data from traffic, weather, positions of my delivery vehicles (from GPS), and incoming orders from my order system.  In a BPM world these events would only come together when they are relevant to a given activity.  In an event driven world, these events could potentially affect each other at any time, regardless if there is an active process task or not.  Many of these might not even be part of the BPM system.  When the filtered events are correlated to something interesting, a process action can be signaled. Examples of process action include starting a new process for corrective action.  You could also interrupt, continue, or end a process that is in-flight.

The over-all strategy of CEP + BPM means that there is less integration time for systems. Instead of routing all event activity load through the BPM system, the CEP system handles complex correlations of events, and aggregates them down to things that a BPM system might be interested in.  Also, you can get a broader perspective of multiple process instances simultaneously, especially when activities are in-flight.  For example, a delivery truck is running late.  With BPM only, it’s likely that you would not know this fact until the truck is overdue.  With CEP, I can constantly monitor all trucks via their GPS signal, and report all positions to a dashboard. Furthermore, you can correlate to traffic data in-route, and determine exactly how late the truck will be.  This allows an organization to react to exceptional conditions in real-time.   Keeping in mind, most transport companies (and other industries) do this sort of monitoring anyway, but not automatically.  Without CEP + BPM capability, you would have to perform these correlations manually, which might require a huge workforce.  The key here is that the events are reported in real-time, and the event correlations are more accurate.

Included in this tour: Atlanta, Chicago, Vancouver, and then wrapping it up in San Francisco.    The audience was fairly broad, including project managers, IT engineers/architects, business analysts, and even a few executives.    There are several things I found in common from the questions at each of these sessions.

1. The IT trends are always changing, so why should I start now with BPM and SOA?

That’s a great point.  Interestingly I get this question every time I present this topic.  Yes, technology trends are always changing. But this is something different.  Previously the right elements were not all in place to fully take advantage of the BPM paradigm.   Today many BPM software vendors are creating SOA based products, and SOA software vendors are creating “orchestration” products which are very similar to BPM concepts.  Basically out-of-the-box you get a whole lot of BPM and SOA from one product.  Previously (2002 – 2005) this would have required several layers of middleware and custom connectors to achieve.  So the best way to say this is that now BPM and SOA are consumable by an organization.   It’s not quite to the point yet where the systems are such a commodity that it’s like a word processor.  But even the common word processor is starting to have some collaboration capability built in.  So where do you think this is going?   In my opinion I think we are past the point of experimenting with BPM and SOA.  It’s now become common practice in many organizations and has enough financial backing due to the customer demand that it’s here to stay.

While in my Vancouver session one of my attendees asked this same question in a way that I found interesting.  Their organization had worked with various case tools and workflow automation tools over the years and they said that they have seen many “code generation” tools that have made the same claims as BPM promises to deliver but they never did.   So it seems that the motivation for the skepticism is based on slick sales people offering that classic “80% out of the box solution” that you can “fire your IT people and do it yourself”.   Well, you’ll never hear me telling you to fire your IT people.  You might need to educate them on process modeling, but you still need them to create good SOA components for you.

The BPM vendors of the early decade (2002 – 2005) are partially the cause of the skepticism of the BPM potential.  It used to be a hot buzzword back in 2003 but it went away for a while before coming back strong in 2008.  Often a vendor would try to sell BPM to an organization.  Who would have known that you have to actually do BPM, and you can’t actually buy it?  Well, times have changed and it’s time to take a second look at what’s available.

Another reason why I see it’s time to get into BPM and SOA is that organizations tend to look at ways of being more efficient in times where cash is tight.  In times of plenty, we just hire more warm bodies to throw at a problem.  Later we can just fire them, right?   Or on the other hand, we can learn to manage our processes and systems in a more efficient way so that we don’t have the huge downfall after times of rapid growth.   After a period of recession there will be a period of growth.   After you’ve already grown out of control it’s hard to get a handle on the problem.   So maybe you can  get started now before you miss this window of opportunity.

2. What are the most common problems I see in implementing BPM?

This one is easy.  There is not a lot of BPMN process modeling skills in most organizations.  The people that I find who know how to model a process well are usually consultants such as myself.    A basic BPMN training course is a good start.  But after the basics of BPMN are learned additional coaching/consulting is likely needed to do a full scale project.   Process modeling for the purposes of automation/execution is much different then modeling the ‘as-is’ process.  Different patterns are used, and sometimes these patterns are vendor specific for the BPM automation system.  So it’s one thing to model your ‘as-is’ process.  Learning how to make it actually work is another skill that you must learn by practice.

So the short answer to the question of what are implementation challenges is that there is no widely practiced methodology that works well with both SOA and BPM.  Waterfall?  Forget it.  Scrum, Agile?  Sort of, but not quite.  Test driven?  Mmm.. partially.    PMF includes the best of all the above, and is totally oriented towards the business objectives and process flow.   On one hand you have methodologies that are totally focused on data models and on the other you have high-level flowcharts.  In the middle there is really nothing to bridge the gap.

For this reason I have created the Process Modeling Framework (PMF).  PMF was briefly introduced in my book The Microguide to Process Modeling in BPMN but we did it no justice.  The book’s title says “microguide” so please don’t’ be disappointed if the concept is a bit thin in this book.   The intention of the book is to give you a quick start on a consumable, non-technical format.   But for those of you who want to learn more I’m in the process of writing the next book that will cover PMF in great detail.

3. What are the most common challenges I see in implementing SOA?

Most of the time I see engineers creating data models as the first step to a system design.  This I see as the greatest roadblock.   What you have to understand is that data would not exist if there was no process first.   What is the point of having data?   Do you just create data as the objective of the process?  Or is there a reason for doing so?   In other words, stop modeling data and start modeling processes.   The process will dictate what the data requirements are, and not the other way around.

I realize this is nearly treason for some of you on the SOA side of the fence.   To the more technically minded the data is everything.  And I totally agree with you that data is important.  But let’s not get into a “chicken or the egg” debate.  I you already have data, manage it, but in a way that supports the business objective.  Don’t start going out of your way to create an SOA infrastructure when you don’t even have a solid process model to back your assumptions.  Otherwise you might end up making the situation worse by creating an architecture that restricts the organization’s capability to grow.

4. How do I handle developing many services and many subprocesses in parallel?

For the most part you need to limit the scope of diagrams to a single business objective.  Stop creating flow charts that wrap around the room.  Stop modeling everything in terms of swimlanes.  There is more going on here than one person passing a task to another person.   Any process in a large organization likely interescts with other processes.  So maybe what you should be drawing is how the processes interect with each other, and not how people interact.  People are part of the organization, and the organization is made up of processes.   This all relevant when determining what is the scope of a diagram.  Once you start mixing objectives from processes that overlap/intersect/conflict with each other, then there is no way you can have multiple people work on it.  It’s like saying that you are going to take 10 race car drivers, give one the steeing wheel, another to the brakes, another to the acellerator, etc.  It’s just not going to work this way.  Every driver needs to have their own car.    So stop trying to model everything as if there is only one car with many lanes (swimlanes).  Instead there are many cars, with many lanes, and not all of them flow in the same direction.

5. How much of the BPM/SOA methodology should I teach to the business stakeholders?

At first, as little as possible while still allowing yourself to do your work.  Only explain as much as what is necessary to get the information you need from the stakeholders.  Otherwise you’ll get into nearly religious debates about methodologies that evolved from the 1970′s.  Gradually as you show results, the organization will adopt BPM/SOA because of the obvious gains in productivity, visibility accountability, and agility.   So instead of upsetting the entire organization overnight, start slowly and show results, not radical concepts.  Results in process modeling come from showing how to improve the organization.  Results in SOA come from showing how quickly you can create a composite system from several existing components.

Today we have accelerated expectations yet we still use methodologies based on a time when computers were the size of refrigerators and access to a computer was only for those lucky enough to have one in their office.   Yes, you are trying to overcome this problem.  But you are fighting an uphill battle with “traditional wisdom”.   Traditional wisdom doesn’t have a 3 GHz quad core CPU with 8 GB of RAM, connected to the internet at 10 gigabits per second.   But I do, and you probably do too.   Computers are just as much a part of the organization as its people are.   90% of corporate computer systems  are sitting idle 90% of the time because we still haven’t figured out to efficiently use them yet as part of our business process.   We’ve got way more computing power than we know what to do with, yet we still pass around pieces of paper or spreadsheets over email.

Let me ask you this:  When is the last time you emailed a spreadsheet to a group of people and collected responses?  Sorry, but this is not a process.  It’s chaos in the making.  Yet a majority of the organizations I work with use this technique of information mis-management as common practice.   They know there is a better way, but at the same time they resist changing.   After all, if it isn’t broke, why try to fix it, right?   I’ve got a reason of you.   It’s called financial meltdown – just  incase you haven’t been watching the news lately.

The older methodologies never took this phenomenon of high speed computing into consideration because it didn’t exist yet.   The world has changed dramatically in the past decade.  Now I can connect to people via my “video phone” in a virtual world and I can book my own travel tickets faster than I can explain to someone where I want to go.   Or better yet, I can schedule a trip to Chicago on my calendar and my saved preferences will automatically book my flight for me, and all I have to do is confirm it.   This is an example of process oriented, as opposed to the way it’s always been done.

You have to consider that you cannot change the mindset of “the machine” of your large organization overnight.  Some very respected people work in this same organization as you have built their reputation on these older methodologies.   One tiny voice shouting BPM and SOA will not be heard over a hundred entrenched opinions based on traditions.   By going on a crusade to change everything, all you are doing is shaking the tree of traditional wisdom, and whatever falls out of this tree is likely to come down hit you hard on the head.    Many of these very respected people still call a travel agent over the phone and use a paper ticket at the airport.   On the other hand, many of them ride around in a corporate jet and haven’t seen an airport security line since 2000 when it was still okay to meet your friends as they walk off the plane at the gate.

If you are lucky enough to have a CIO or CEO that mandates the organization will have a BPM/SOA center of excellence team (CoPE team) then you can go ahead and start a formal training program.  But still, the details will be missed by most people in the organization.   People tend to do 5 to 7 activities in their processes.  Outside the scope of what they do on a day to day basis is hard to grasp.    The bigger picture of a formal BPM/SOA practice is likely to be well received to a C-level executive, a VP, a director, etc.  But anyone lower than this and it’s likely they’ll need a Business Analyst or IT architecture background to understand where you are going with these concepts.

But the good news is that over time as the organization transforms its self, every one else will eventually understand by seeing examples of good practices in action.   I’ve seen even the people that have the hardest time changing eventually learn the process oriented approach.  It’s contagous.  And once you go process oriented, you’ll never go back.