In a previous post I wrote about IBM buying Lombardi, and noted that I expect to see more industry consolidation.  Well, we didn’t have to wait long to find out which merger is next.  They say things  occur in threes, so I wonder which BPM company is next….

The IBM acquisition of Lombardi was interesting because IBM needed to fill the department level workflow product need.  This is where Lombardi is being positioned, according to IBM.

Progress software’s move for Savvion was much more interesting because Progress did not have a BPM system, or history of workflow systems.  Instead, Progress has a wide range of SOA systems and tools, and a strong offering in the CEP (complex event processing) and BTA (business transaction assurance) space.   With these offerings, Progress would more likely be positioned to sell products and services to the IT architecture crowd more so than the business.  Now with a full  stack of BPM, CEP, transaction monitoring, ESB, and data services, Progress is very well positioned to be an aggressive competitor in the BPM market.

I’m very interested in this Progress and Savvion merger because it shows the BPM industry is evolving into more event oriented processes instead of people centric activity.   CEP (complex event processing) refers to the idea that things happen that are not necessarily all part of a process.  For example, I can correlate streaming data from traffic, weather, positions of my delivery vehicles (from GPS), and incoming orders from my order system.  In a BPM world these events would only come together when they are relevant to a given activity.  In an event driven world, these events could potentially affect each other at any time, regardless if there is an active process task or not.  Many of these might not even be part of the BPM system.  When the filtered events are correlated to something interesting, a process action can be signaled. Examples of process action include starting a new process for corrective action.  You could also interrupt, continue, or end a process that is in-flight.

The over-all strategy of CEP + BPM means that there is less integration time for systems. Instead of routing all event activity load through the BPM system, the CEP system handles complex correlations of events, and aggregates them down to things that a BPM system might be interested in.  Also, you can get a broader perspective of multiple process instances simultaneously, especially when activities are in-flight.  For example, a delivery truck is running late.  With BPM only, it’s likely that you would not know this fact until the truck is overdue.  With CEP, I can constantly monitor all trucks via their GPS signal, and report all positions to a dashboard. Furthermore, you can correlate to traffic data in-route, and determine exactly how late the truck will be.  This allows an organization to react to exceptional conditions in real-time.   Keeping in mind, most transport companies (and other industries) do this sort of monitoring anyway, but not automatically.  Without CEP + BPM capability, you would have to perform these correlations manually, which might require a huge workforce.  The key here is that the events are reported in real-time, and the event correlations are more accurate.

Application or Process?

A business process exists whether or not you automate it through a BPM system or a workflow tool.  Many organizations choose not to use a formal BPM approach to process.  Instead they use the traditional vertical market application that helps automate some of the process based on rules and logic provided by the software vendor.  Some degree of process management exists with this approach.  However the logic (the business know-how) is essentially outsourced to the software vendor.  Often this requires the organization to undergo a massive customization effort in order to make the vertical market solution fully effective in the organization.  

The application acts as a participant to the process.  Without there being a business process there would be no need for the application.  So you could say that what you use the application for is the process, and the application is the tool that helps you be more efficient at doing your part of the process. 

Why BPM is a constant

Everything about business involves a process.  Presenting a product to a customer, ordering supplies, and collecting money are all examples of processes.  In a more simple term you could all each of these activities a workflow.  The real benefit of BPM comes into play when you start to analyze the complex interaction between many of these individual workflows.  Most likely the simple workflow evolves to include computer systems, probably just simple applications at first, gradually becoming more complex.  These systems become participants of the process as well.  Eventually the computer systems become an integral part of the process, often automating parts of the original process as well as enabling more efficiency as more people and systems are involved.

Let’s not forget that BPM is a management technique more than it is about technology.  With BPM we are not talking about managing specifically people, systems, vendors, customers, or money.  Instead we are talking about managing people, systems, vendors, customers, and money, as well as the complex interactions between them.  The more complexity exists in a process the more efficient your organization becomes with proper business process management methodology and technique.

The next trend in IT

Enough about defending the need for BPM.  Here’s what I see as the next emerging trend in information technology: Could Computing. 

So what does cloud computing have to do with BPM?  A lot, actually.  Earlier in this post I contrasted the difference between applications and process.  But we are now at a point in history where this line becomes even more blurred.

At the core of cloud computing is hardware; a lot of it. But instead of 4 or 5 servers to run one application we start to see a trend where 10 servers run 40 or 50 applications.  The hardware virtualization means that CPUs, memory, and storage capacity is combined over multiple systems.  The old days of redundant systems for maximum fault tolerance are coming to an end because the basic architecture of a cloud system is 3 and 4 times redundant in every way.  In fact I’ve seen demonstrations where the plug is literally pulled out of the wall and the system keeps running.  This is because there are dozens of power supplies with dozens of plugs, and often multiple complete systems in different data centers, all acting as one gigantic supercomputer. 

The could computing design is more reliable, and it’s cheaper to operate.  Moore’s law states that computer power doubles about every two years.  The problem in data centers is that in most cases 95% of the CPU power is wasted sitting idle because it’s only utilized when someone is using that specific system.  When people happen to be using another system, the CPU simply burns up kilowatts of power from the electric company as it remains idle, waiting to process the next request from a user; as quickly as possible.   But with cloud computing, a cluster of computers are all operating as one.  A peak load on one application can easily be absorbed across the entire system while the less frequently used applications.

The idea of cloud computing is undoubtedly inspired by the way the Internet works.  At any specific point on the backbone of the Internet, if a failure occurs, there might be a localized disruption. But nobody has ever heard of the entire Internet going offline.  It’s designed to be fault tolerant to a point where even nuclear war won’t take it offline.

Now back to my point about where BPM is involved in all of this.  Prior to the cloud computing trend, systems were isolated in various localized data centers with virtually no way to communicate with each other besides for the system interfaces (API) that were designed to perform a specific function.  Cloud computing brings systems previously separated by physical hardware together on one hardware platform.   When you add service oriented architecture (SOA) to this combination, there starts to be less objections to using web services.  Many software engineers and architects believe that a native interface (such as Java to Java or .NET to .NET) is better than web services for performance reasons.  But in a Cloud environment when an application needs to talk to another system, often the other system is in the same virtual memory space on the same hardware cluster.   

No longer do we have to worry about databases exceeding 50 terabytes. The cloud system can handle exabytes or even petabytes without even a flinch.  So the notion of storing data once in a “normalized” database becomes too much effort to make it worth the effort of doing proper data modeling.  Store it 1000’s of times in 100’s of formats to service dozens of applications because you can transform the data just as fast as you can store it.

Again, BPM is about management.  If I can connect everything, nobody objects to connecting, and I have a virtually unlimited amount of computing power, imagine the complexity we can create!    So what are you going to do with it?  Burn up megawatts of power instead of the kilowatts of previous generations?  Instead, how about getting smart about what you do with your computing power.  All of the computing power in the world will not do anything for your organization if you don’t manage the processes that you are attempting to automate. 

The trend as I see it will be for applications to be more process aware, and BPM systems to become more like applications.  Many early attempts at BPM systems were nothing more than task state management for workflows.  System to system integration was added, but this is still very task centric.

There are several categories of applications.  Some examples include:

• Tools such as a calculator, disk defragmenter, backup utility, etc.
• Data origination tools such as a word processor, spreadsheet
• Information sharing tools such as email, screen sharing, etc.
• Collaboration tools such as groupware, and BPM automation systems.

The problem is that we still need one application for one job, and another application to do something else.  For example, my word processor program is good for writing documents but doesn’t do so well at adding 2 + 2.  My spreadsheet crunches numbers well but doesn’t manage people well (although some people insist that a spreadsheet is actually a database).

History hints at what’s next

If there is anything that history tells us about technology, it’s that consolidation of multiple systems is inevitable.  Back in the 1970’s a CB radio or walkie talkie was all the rage in business communications.  Then someone got the idea to combine a radio with a telephone and the cellular telephone was born.  While we are at it, why not put a camera on it.  Personally I couldn’t figure out this marriage of technologies  out when it first emerged, but now I find myself sending pictures of my daughter to my friends and family on a regular basis.   Oh, and while we are at it, why not hook the phone to the internet.  For that matter, why not hook your refrigerator and toaster oven to the internet too?   That way you can call up your appliances and tell them and make you breakfast before you get out of bed.     Or better yet, as I sit here stretched out in business class on my favorite airline, I’m writing a blog post while connected to the Internet, powered by cellular phone technology.

In the above example there are a few major enablers of the merging technology.  First there is a reliable cellular telephone network that is available virtually anywhere on earth people are found in mass.  Next there is the Internet; always on; always ready to serve.   This is the infrastructure.  The telephone and the camera are the tools.  The collaboration is when I hit the send button from 30,000 feet (9500 meters) above sea level, telling my wife how wonderful the remote control for the powered lay-flat seats are on this airliner, and she replies saying “that’s nice honey, enjoy. I have to put the little one to bed”.   I’m constantly in touch with the world, powered by so much complexity that I never have to know about, or even know that the complexity exists.  

The cloud , simply put, creates an enabling infrastructure that I never have to worry about.  It’s there, it’s always on, and it would take a full-blown nuclear war or an asteroid hitting the earth to take it offline (in which case we’d all be dead anyway, so why worry about it).  Data exists somewhere, but I no longer care where it is.   Someone just added 5 more terabytes of RAM to the cloud and I didn’t even know it (yes, I said terabytes of RAM, not hard drives).  My software got updated with a click of the button, and if I don’t like it I can click a button to switch back to the previous version without the tedious process of uninstall, reinstall.

So I’m on the internet writing on my blog page (the cloud enabled application) writing about the cloud, while I’m looking down at the clouds.   Sorry to mention it, but I couldn’t resist pointing out the irony.

The green screen effect

It’s been said many times that we are coming full circle back to the days of the green screen.  Ironically ‘green’ means something else today, which is causing the push to go back to the concept of the terminal attached to the mainframe.  For those of you reading this who are too young to remember, the terminal was a green CRT screen.  Remember the CRT?  You know, that huge clunky tube screen.   Before they CRTs were colored they displayed characters either green or yellow.  Green today means using less power and being friendly to the environment.  One of the most compelling arguments for moving to cloud computing is because it’s more environmentally friendly as well as easier to manage.   It also means distributing computing power everywhere like a grid, and hosting applications online instead of installing them on your local machine. 

Recently Google announced that they are releasing an operating system that is not much more than a window to the Internet.  Anyone see where this is going yet?   If you think about my example of how the cell phone enabled me to make toast in the morning without getting out of bed, what happens when the Internet meets cloud computing and both applications and processes are so intertwined that you can’t tell where one stops and the other begins?   Applications?  Process?  Don’t know, don’t care.  I have work to do so stop bothering me with such trivial things.   

I don’t have to install applications anymore.  I simply have to cache the data locally incase by odd occurrence that I cannot connect to the Internet.   Even the green screen terminals of the 1960’s had this concept.  They “buffered” the data in an 8 kilobyte local microchip in case the connection to the mainframe was severed momentarily.  Well, the numbers certainly got bigger, but the concept is coming full circle.  Want a word processor or spreadsheet?  Try Google Docs.  Want an image editor?  Yep, you can do this online too.   Why keep it local?  Local storage is a single point of failure that packrats like me who never delete anything cannot afford to risk.    Even this blog post is auto-saved out to the cloud somewhere and I don’t have to worry losing anything even if this plane I’m sitting on crashes.   Google’s applications automatically store a temporary copy locally until the data is sent to the cloud for permanent storage.  I bet some of you didn’t even know it works this way.  That’s the point.  It’s fault tolerant, it’s green, it’s empowering, and it’s transparent.  At least to me, this sounds like a trend that is here to stay.

Person, system, or process?  They are all process, out in the cloud.  It won’t be long and I’ll be having a conversation with your virtual presence or avatar while you are out of the office.   I’m hoping people will finally realize that sending spreadsheets over email is not process management – it’s completely wasteful.  While we are at it, let’s get rid of email too and think of something more efficient because I’m tired of sifting through 200+ emails per day trying to follow a conversation.  And this way I can ensure I never get another spreadsheet emailed to me. Please link it, don’t send it. This way I can actually find the correct version when I need it instead of sifting through 5 or 6 outdated versions.

Getting to the point

In case you don’t see where I’m going at this point with this post, let me spell it out very clear and concise.   Show people they can things done without worrying about the technology that powers it and you will start to see things getting done.  Take the complexity away from the worker and give them what they need, when they need it.  Make it convenient and available wherever, whenever.   Don’t bog them down with the complexity of technology. Likewise, don’t bog down the workforce of your organization with the complexity of business processes.  They don’t need to know how it works, just that it does.  Make it simple, seamless, and bullet-proof reliable.  Don’t make them worry about whether or not the data is accurate and up to date.

So the trend I see for the future of BPM is that a new wave of process management will emerge that is not bogged down by legacy fears and horror stories of integration challenges.    Applications that you install so support the business process will become merely windows to the data produced by the instances of processes. Synchronizing data to the local machine in a “buffer” will become a standard feature for all BPM systems, and local data storage will likely become the backup copy rather than permanent storage.  The permanent storage is somewhere on the cloud but it will be nearly impossible to determine the precise physical location where it exists.   In many ways, the technology world as we’ve known it for the past decade is taking a full reversal and going back to the concepts of the 1960’s.   But Moore’s law of computing power will actually accelerate.  And along with this, I believe BPM will become hundreds of times more effective when practiced religiously throughout the organization.  Before this can happen I think that BPMN will have to evolve to deal with more complex event processing needs because the complexity will certainly be there, even if most of us no longer have to know the complexity exists.

You might have noticed that I haven’t written any posts in a while.  The reason is because I had an epiphany.  A light bulb turned on in my head (and yes, it was a compact florescent, because I’m on the green trend too).  It was such an crazy thought that I started to wonder if my passion for BPM is going somewhere.  Then just at my most critical moment, a dear friend asks me if I’m still doing “that process stuff”.    I stand by my words.  Yes, I do BPM.  And I do it because there is more of a need for it now in the cloud computing age than ever before.   Those of us who specialize in process management need to realize that we have invented a profession.  We are not IT specialists, business analysts, efficiency experts, or project managers.  We are a bit of all of these things in one.  This is how I got the job title Process Expert, and I’m looking forward to a time in the not-so-distant future that this will be a job title I can find on the job search sites on a regular basis.

I see that this post is getting rather long so it’s time to wrap it up and open it for discussion.  I’m looking forward to getting comments from my readers so that I can write more on this topic.  But for now It’s time for me to do a video conference with my 11 month old child from 37,000 feet up in the air (camera + laptop + internet + airborne internet was a wonderful marriage of gadgets).  I wonder what she’ll be doing when she’s my age?   One thing is for sure, she’ll be online.  Someone will probably have found a way to remove the thunderous background jet noise from the airborne video call.  But I doubt by that time the FAA (Federal Aviation Administration for those of you outside the USA) will allow her to stay online during takeoff and landing.   Until then, there is a lesson to be learned.  Never forget to model the exceptional conditions into your processes, no matter how reliable the underlying technology becomes.

Included in this tour: Atlanta, Chicago, Vancouver, and then wrapping it up in San Francisco.    The audience was fairly broad, including project managers, IT engineers/architects, business analysts, and even a few executives.    There are several things I found in common from the questions at each of these sessions.

1. The IT trends are always changing, so why should I start now with BPM and SOA?

That’s a great point.  Interestingly I get this question every time I present this topic.  Yes, technology trends are always changing. But this is something different.  Previously the right elements were not all in place to fully take advantage of the BPM paradigm.   Today many BPM software vendors are creating SOA based products, and SOA software vendors are creating “orchestration” products which are very similar to BPM concepts.  Basically out-of-the-box you get a whole lot of BPM and SOA from one product.  Previously (2002 – 2005) this would have required several layers of middleware and custom connectors to achieve.  So the best way to say this is that now BPM and SOA are consumable by an organization.   It’s not quite to the point yet where the systems are such a commodity that it’s like a word processor.  But even the common word processor is starting to have some collaboration capability built in.  So where do you think this is going?   In my opinion I think we are past the point of experimenting with BPM and SOA.  It’s now become common practice in many organizations and has enough financial backing due to the customer demand that it’s here to stay.

While in my Vancouver session one of my attendees asked this same question in a way that I found interesting.  Their organization had worked with various case tools and workflow automation tools over the years and they said that they have seen many “code generation” tools that have made the same claims as BPM promises to deliver but they never did.   So it seems that the motivation for the skepticism is based on slick sales people offering that classic “80% out of the box solution” that you can “fire your IT people and do it yourself”.   Well, you’ll never hear me telling you to fire your IT people.  You might need to educate them on process modeling, but you still need them to create good SOA components for you.

The BPM vendors of the early decade (2002 – 2005) are partially the cause of the skepticism of the BPM potential.  It used to be a hot buzzword back in 2003 but it went away for a while before coming back strong in 2008.  Often a vendor would try to sell BPM to an organization.  Who would have known that you have to actually do BPM, and you can’t actually buy it?  Well, times have changed and it’s time to take a second look at what’s available.

Another reason why I see it’s time to get into BPM and SOA is that organizations tend to look at ways of being more efficient in times where cash is tight.  In times of plenty, we just hire more warm bodies to throw at a problem.  Later we can just fire them, right?   Or on the other hand, we can learn to manage our processes and systems in a more efficient way so that we don’t have the huge downfall after times of rapid growth.   After a period of recession there will be a period of growth.   After you’ve already grown out of control it’s hard to get a handle on the problem.   So maybe you can  get started now before you miss this window of opportunity.

2. What are the most common problems I see in implementing BPM?

This one is easy.  There is not a lot of BPMN process modeling skills in most organizations.  The people that I find who know how to model a process well are usually consultants such as myself.    A basic BPMN training course is a good start.  But after the basics of BPMN are learned additional coaching/consulting is likely needed to do a full scale project.   Process modeling for the purposes of automation/execution is much different then modeling the ‘as-is’ process.  Different patterns are used, and sometimes these patterns are vendor specific for the BPM automation system.  So it’s one thing to model your ‘as-is’ process.  Learning how to make it actually work is another skill that you must learn by practice.

So the short answer to the question of what are implementation challenges is that there is no widely practiced methodology that works well with both SOA and BPM.  Waterfall?  Forget it.  Scrum, Agile?  Sort of, but not quite.  Test driven?  Mmm.. partially.    PMF includes the best of all the above, and is totally oriented towards the business objectives and process flow.   On one hand you have methodologies that are totally focused on data models and on the other you have high-level flowcharts.  In the middle there is really nothing to bridge the gap.

For this reason I have created the Process Modeling Framework (PMF).  PMF was briefly introduced in my book The Microguide to Process Modeling in BPMN but we did it no justice.  The book’s title says “microguide” so please don’t’ be disappointed if the concept is a bit thin in this book.   The intention of the book is to give you a quick start on a consumable, non-technical format.   But for those of you who want to learn more I’m in the process of writing the next book that will cover PMF in great detail.

3. What are the most common challenges I see in implementing SOA?

Most of the time I see engineers creating data models as the first step to a system design.  This I see as the greatest roadblock.   What you have to understand is that data would not exist if there was no process first.   What is the point of having data?   Do you just create data as the objective of the process?  Or is there a reason for doing so?   In other words, stop modeling data and start modeling processes.   The process will dictate what the data requirements are, and not the other way around.

I realize this is nearly treason for some of you on the SOA side of the fence.   To the more technically minded the data is everything.  And I totally agree with you that data is important.  But let’s not get into a “chicken or the egg” debate.  I you already have data, manage it, but in a way that supports the business objective.  Don’t start going out of your way to create an SOA infrastructure when you don’t even have a solid process model to back your assumptions.  Otherwise you might end up making the situation worse by creating an architecture that restricts the organization’s capability to grow.

4. How do I handle developing many services and many subprocesses in parallel?

For the most part you need to limit the scope of diagrams to a single business objective.  Stop creating flow charts that wrap around the room.  Stop modeling everything in terms of swimlanes.  There is more going on here than one person passing a task to another person.   Any process in a large organization likely interescts with other processes.  So maybe what you should be drawing is how the processes interect with each other, and not how people interact.  People are part of the organization, and the organization is made up of processes.   This all relevant when determining what is the scope of a diagram.  Once you start mixing objectives from processes that overlap/intersect/conflict with each other, then there is no way you can have multiple people work on it.  It’s like saying that you are going to take 10 race car drivers, give one the steeing wheel, another to the brakes, another to the acellerator, etc.  It’s just not going to work this way.  Every driver needs to have their own car.    So stop trying to model everything as if there is only one car with many lanes (swimlanes).  Instead there are many cars, with many lanes, and not all of them flow in the same direction.

5. How much of the BPM/SOA methodology should I teach to the business stakeholders?

At first, as little as possible while still allowing yourself to do your work.  Only explain as much as what is necessary to get the information you need from the stakeholders.  Otherwise you’ll get into nearly religious debates about methodologies that evolved from the 1970’s.  Gradually as you show results, the organization will adopt BPM/SOA because of the obvious gains in productivity, visibility accountability, and agility.   So instead of upsetting the entire organization overnight, start slowly and show results, not radical concepts.  Results in process modeling come from showing how to improve the organization.  Results in SOA come from showing how quickly you can create a composite system from several existing components.

Today we have accelerated expectations yet we still use methodologies based on a time when computers were the size of refrigerators and access to a computer was only for those lucky enough to have one in their office.   Yes, you are trying to overcome this problem.  But you are fighting an uphill battle with “traditional wisdom”.   Traditional wisdom doesn’t have a 3 GHz quad core CPU with 8 GB of RAM, connected to the internet at 10 gigabits per second.   But I do, and you probably do too.   Computers are just as much a part of the organization as its people are.   90% of corporate computer systems  are sitting idle 90% of the time because we still haven’t figured out to efficiently use them yet as part of our business process.   We’ve got way more computing power than we know what to do with, yet we still pass around pieces of paper or spreadsheets over email.

Let me ask you this:  When is the last time you emailed a spreadsheet to a group of people and collected responses?  Sorry, but this is not a process.  It’s chaos in the making.  Yet a majority of the organizations I work with use this technique of information mis-management as common practice.   They know there is a better way, but at the same time they resist changing.   After all, if it isn’t broke, why try to fix it, right?   I’ve got a reason of you.   It’s called financial meltdown – just  incase you haven’t been watching the news lately.

The older methodologies never took this phenomenon of high speed computing into consideration because it didn’t exist yet.   The world has changed dramatically in the past decade.  Now I can connect to people via my “video phone” in a virtual world and I can book my own travel tickets faster than I can explain to someone where I want to go.   Or better yet, I can schedule a trip to Chicago on my calendar and my saved preferences will automatically book my flight for me, and all I have to do is confirm it.   This is an example of process oriented, as opposed to the way it’s always been done.

You have to consider that you cannot change the mindset of “the machine” of your large organization overnight.  Some very respected people work in this same organization as you have built their reputation on these older methodologies.   One tiny voice shouting BPM and SOA will not be heard over a hundred entrenched opinions based on traditions.   By going on a crusade to change everything, all you are doing is shaking the tree of traditional wisdom, and whatever falls out of this tree is likely to come down hit you hard on the head.    Many of these very respected people still call a travel agent over the phone and use a paper ticket at the airport.   On the other hand, many of them ride around in a corporate jet and haven’t seen an airport security line since 2000 when it was still okay to meet your friends as they walk off the plane at the gate.

If you are lucky enough to have a CIO or CEO that mandates the organization will have a BPM/SOA center of excellence team (CoPE team) then you can go ahead and start a formal training program.  But still, the details will be missed by most people in the organization.   People tend to do 5 to 7 activities in their processes.  Outside the scope of what they do on a day to day basis is hard to grasp.    The bigger picture of a formal BPM/SOA practice is likely to be well received to a C-level executive, a VP, a director, etc.  But anyone lower than this and it’s likely they’ll need a Business Analyst or IT architecture background to understand where you are going with these concepts.

But the good news is that over time as the organization transforms its self, every one else will eventually understand by seeing examples of good practices in action.   I’ve seen even the people that have the hardest time changing eventually learn the process oriented approach.  It’s contagous.  And once you go process oriented, you’ll never go back.

Topics of discussion include:

• The process oriented methodology for software development, which yields software that is more in-line with the business objective and is more agile for the ever-changing business need.
• How to build software for business requirements that are a constant “moving target” (ever changing).
• Better communication between IT engineers and the business units through model driven software design in BPMN.
• Techniques, tips, and tricks to deliver your software to your customer sooner, and to meet more of the business objective with the product you deliver.
• BPMN is not just for business people – it’s capable of doing some very technical things as well.
• How BPEL (Business process execution language) relates to service oriented architecture, and more specifically how it relates to Java development.
• The modern architecture of process governance, which includes SOA and BPM and how this relates to your job as a software architect/engineer.

Application: Something you install on your hard drive that was designed to help you be more productive. Usually it’s some kind of tool such as a word processor or spreadsheet. Applications can also be online through a web browser, but this is just a difference of how it’s distributed and viewed. Usually an application creates data and stores it somewhere. Sharing data requires another application.

Enterprise Software: Something that took a lot of time and money to develop, for a specific purpose, usually in the form of records management. CRM, financial software, and inventory management are all classic examples of this type of software. Most often this type of application is developed as an online system viewed in a web browser. Many users are involved in this type of system. Data sharing and routing is a common feature.

Workflow: Steps that are required to complete a series of tasks in order to achieve a goal. Often multiple people are involved who collaborate on a data artifact. A good example is sending out a meeting request via Microsoft Outlook or Lotus Notes. This is an example of simple workflow that doesn’t need elaborate rules or custom configuration. It’s generic and can be applied to any business situation.

Note: Workflow methodology began in the 1920’s, and yet many organizations still use it to define their software. Interesting.

Business Objective: The reason why you wanted the software in the first place. You know, that thing that keeps making you change your requirements document all the time.

Business Rules: Policy and procedure that is both common practice for your industry and unique to my organization. Ensures consistency across an entire organization.

Business Process: A combination of all of the above. In our BPMN book Tom D. and I define the business process as:

• “A business process is a flow of decision-coordinated activities, conducted by participants and acting on data, information, and knowledge that reach a goal”

So if I have a business process system that is a combination of an application, enterprise software, workflow, my business rules, and ever-changing busines objectives, it looks something like this:

1. A window to the data. This can be online or installed on my hard drive.
2. Data will come in and out of my view.
3. Other people will log into the same system.
4. I will be alerted when there is something new for me to do, or when something interesting is happening that I should look at.
5. The system will handle the routing of my data for me, without the need for me to figure out how to send it, or where to send it to.
6. Much of my decision is automated through well-established business rules.
7. The system is capable of executing workflows specific to my organization.
8. The system will can easily adapt to an ever changing business objective.

The typical “workflow application” might be capable of achieving goals 1 through 5, and sometimes number 6. But 7 and 8 are the most important points, and often the most overlooked. Things that we cannot foresee are usually not included into our design. Therefore, changes don’t exist until they exist. This was fine 15 years ago when things didn’t change much, but today, changes need to occur much faster than ever before.

Usually, a workflow application is coded with certain rules and restrictions. Being flexible enough to accommodate constant change is usually not in the design. Some typical reasons for this inability to adapt include:

• Certain data flow rules and logic is usually created to be specific with an industry
• The original design was based on an idealistic view of the organization without a shred of proof that it will actually work.
• The system was configured to directly mirror the “as-is” workflow without considering the impact of introducing a new system into the ecosystem of the organization.
• The exception workflow is usually never considered. “Oh, that only happens every now and then, and we’ll just deal with that manually”. These are famous last words of every doomed multi-million dollar workflow development project.

I’m not suggesting that every system can account for every unforeseen event in the future. This would be impossible because by definition, unforeseen means that we don’t know about it yet. They key is in the ability to adapt to new requirements as they come. To achieve this goal, try using the following guidelines:

1. Don’t make any assumptions about what it should look like.

Model the business process first, then visualize what will be needed to solve the problem later. Often the mistake is make of creating a data model first, or mocking up some screen shots. Once you have created something and it’s in front of your team, the team will follow your lead and finish what you created. The problem is, most often the business process challenges were not solved before starting to build the foundation. So what you end up with is retro-fitting business requirements into a system before it’s ever completed in the first place.

2. Process models should be kept small enough to manage.

The larger to process diagram, the more complex the system becomes. Often the complexity is unnecessary. Several smaller diagrams tend to keep the system implementers focused on specific objectives, rather than trying to solve the entire problem all at once. Create many diagrams that cover the scope of one scenario or major use case. Link these scenario diagrams together.

3. Use a Service Oriented Architecture (SOA) approach

SOA allows you to create smaller services that are more general purpose, and can be reused in multiple places. Of course, the first time you invent a service there is the investment of creating it properly. But once build properly, it becomes part of a library of lower-level functions that can be consumed by processes. Chaining these SOA components together along with some existing applications and human workflow will get the job done nicely.

4. Limit the scope of your User interface(s)

User interaction is not always a simple form – Sometimes its a miniature application. These days it’s Ok to build a rich UI client application for your process. Technologies such as AJAX made it acceptable. But this does not mean build a big one. This means that for the scope of an interaction, build it as big as it needs to be. But when the scope of a single interaction is completed, the scope of the user interface is also done. Create another user interface for the next interaction task, and reuse components from the previous ones.

5. Don’t spend a lot of time shopping for your solution.

Odds are that you are trying to create a system that fulfills a need unique to your organization. This means that it probably does not exist out of the box. Refer to rule #1 – model it out first. Then you will know what you are looking for. After this, you can make a more educated decision on what you can use from your existing systems, what you need to buy, and what you need to build.

Trust me – it’s all about the business process first. All the other details usually fall into place very easily once you have the correct process model.

For example, an employee submits information to be reviewed by his manager. But how does this happen? Does he physically hand a paper document to the manager? If it’s any type of electronic document, there is a system involved, which should be considered a participant of the process. All participants must be considered in the diagram. Otherwise, pockets of inefficiency exist in your process model, and major details might be completely overlooked when it comes time to re-engineer the process. However, we don’t want to spend time documenting the inner workings of system activities.  This responsibility falls on the IT Information Technology) development team that created the system.  Documenting the internal processes of system activities should be done when a system is created. Therefore, documentation should already exist. As a process modeler, you should focus with how to use this system as a service to the process.

Systems built with the SOA approach are said to be consumable, meaning they are readily available to be used in an automation system. Typically a BPMS is used (business process management system), but it could be any number of other applications that also consume services. Before SOA became popular (around 2005) IT engineers had to write software code called middleware that binds one system to another. The problem with middleware is that it often became a roadblock to progress, because before updating the system on either end, the middleware must also be updated.

Some examples of SOA might include CrUD operations (Create, update, and delete of data), login/authentication services, calculators for common functions such as interest rates or insurance premiums, and common data retrieval such as stock quotes, the weather, etc. Since all of these examples can be generic in nature, a service is created to support a wide range of activity. For example, if I want to get the weather, the service is not tied to any specific location. Instead I must provide the location to the service, and it returns the requested information. An so, often the context of the service activity is left open ended, which allows a service to be created once and used often in many other systems.

In a SOA system, certain parts and pieces are offered as a service to the entire organization. The role of the IT engineer shifts slightly to where their primary responsibility is to create services, and reuse existing assets before purchasing or building new ones. If a service is needed that doesn’t already exist, the first step is to determine if the organization already has a system that performs a similar function. If so, then there are ways to expose an operation (activity) as a service with minimal impact to the surrounding IT environment.

Going back to the process model, I want to send information from an employee to a manager for review. What is actually happening is that the employee is submitting information electronically to a service. The service then determines how to route the information. The service might also perform a query to determine who the submitter’s manager is. If I am modeling this process I might want to stick to the idea of the data submission and what to do with the data when a reply is received from the manager. But the data service should still be part of my diagram.

It is often the case that the data service required by a process model does not exist yet. The process modeler can easily define what is needed for a system to perform the desired activities, such as look up the manager and forward the request. Some of the parts and pieces of this process might already exist. For example, the lookup manager activity might be available as part of the security system. However, we don’t want to build a direct integration into the security system. We just want to leverage some of it’s basic functionality. We don’t want an entirely new system either, because this will likely result in duplication of data that already exists somewhere else.

Figure 1: A system pool shows the requirements for a lookup manager service and a routing service
Figure 1 is a simple BPMN diagram showing the need for a request router service. When the IT engineering team sees this diagram, they should immediately spot the need for two services. 1) The request router. 2) the lookup manager service. The lookup manager service would probably be not be directly used by the process. Instead, we would use the lookup manager service as part of the request router. The request router would most likely be created as a generic service that could also handle the manager response. Since the submitter is already known, we don’t need to query for him. The request router service would simply forward back to the submitter.

The process model in figure 1 is very simple, but might be very valuable. After further developing this diagram with good text annotation, data artifacts, and good labels, we can replace an equivalent requirements document that might have been 10 – 20 pages in length. Provided that your IT organization understands BPMN, it’s very easy to determine system use cases from this diagram. Each scenario we model might create another use case. For example, as part of the first scenario of submitting to the manager we created the diagram in figure 1. When we add the manager’s reply to the service requirements, the diagram might look like figure 2.

Figure 2: Diagram refactored to include the reply from manager

From an IT perspective, the service is created to route messages in a generic way. If I have an address I send it directly to the person. Otherwise, I have to determine the person to route the request to. The service could be created with operations such as “Send to person jdoe@mycompany” and “Send to my manager”. In the process we automatically know now to send something to anyone’s manager by consuming this service.

This approach to managing processes and services is very efficient for both the business analyst that models the processes and the IT engineer/architect that creates and maintains the services. In fact, the role of both IT and BA becomes much easier, and better, more accurate requirements documents result. This method promotes collaboration between IT and BA staff rather than putting up walls of responsibility. The IT staff is more efficient because instead of creating a huge, monolithic application that takes years to implement, they are only creating one more service that is compatible with their infrastructure. On the business analyst (BA) side, you don’t have to spend weeks or months negotiating with the IT staff on what can and cannot be delivered in time. Much of the functionality might already exist, and what needs created is very small in scope so it shouldn’t take as long as the traditional software development approach.

In summary, SOA and BPM need each other.  BPMN is the language that bridges the gap between the business process world and the IT world.   IT creates services that have high-level documentation in BPMN that is easily readable by business people.  Business analysts create diagrams in BPMN that are easy for IT to interpret into technical requirements.