Tuesday, April 29, 2008

What we learned about EA implementations

(based on many people asking why I recommend the implementation approach I do)

The fact is that most EA implementations fail i.e. they fail to produce sustainable value and a useful asset i.e. one that can be used to: answer questions (e.g. impact analysis), maintain knowledge, support business operations and business change, etc. Over almost two decades I have seen dozens fail (often undertaken by some of the brightest, most able, and most diligent people I have met).

I have seen EA initiatives in many types of organisations (forestry, health, insurance, government, finance, health and safety, eCommerce, retail, education, telecommunications, etc) and using many different generic frameworks and methods (e.g. Zachman, TOGAF, etc.) - and the quality of the result does not seem to be associated with either the type of organisation or which of these generic approaches is used.

It is true that in many cases inadequate tooling is the root cause, and EA "consultants", "experts" are quite happy to continue using office documents and essentially manual approaches because for service companies it is far more profitable to do the fishing, than sell a rod and teach the person how to fish for themselves.

Where poor tooling is not the cause what I usually have seen is work on an EA implementation commence:
  • lead by people who only understand about 10-15% of what the technology being used can do (and have a partial understanding of its intended use) or lead by people who understand the tooling reasonably well but who have very limited domain expertise (i.e. in enterprise architecture, ICT strategy, business/IT transformation etc.).
  • without a clear understanding of what is sought (requirements)
  • without a clear understanding of how the solution will be implemented (all the components, how design will be undertaken, how operational procedures will be defined etc.)
  • without a well defined (tried and tested) implementation plan
  • without an understanding of the organisations change impedence issues (the organisational behaviours that impede change) associated with improving touchpoint processes and roles.
Unsurprisingly these implementations seldom proceed well and usually erroneous conclusions drawn from what is delivered (from those who "don't get it" or sometimes "don't want to get it").

If one doesn't start with a clear view of the outcomes being sought - it is unlikely the benefits will effectively be achieved (i.e. if you don't understand the requirements it is unlikely the solution you will deliver something that achieves them).

Therefore what is required upfront is a focus on what outcomes should be sought, i.e.
  • who are the stakeholders and/or beneficiaries ?
  • what are the scenarios of use they have ?
  • what are the business questions they need answered ?
  • and in roughly what stage of an implementation ?
Some EA approaches aim to achieve this but they are so generic (i.e. they don't make an assumption that suitable tooling will be used) so they can't be precise enough about the nature of the requirements captured, or how the design will proceed.

This is why we have developed a detailed implementation approach. This defines all the major areas of work, the key roles, logical milestones etc. that is specific to the implementation technology we use and can be instantiated quickly for a specific project

People often think that there must be a single detailed approach to developing an EA that will suit every organisation. There is a common meta-approach - but the detailed approaches differs based on the organisation and the goal. The problem is that different organisations are focused on making different types of changes e.g. new products or new geographies, mergers, risk reductions (including regulatory compliance, DR, BCP), cost reduction etc. so their focus is naturally different e.g. I have seen EA implementations with many different orientations e.g.
  • Business and/or technology strategy (usually aiming at getting an explicit consensus as to what changes should be initiated and why).
  • Business process redesign (adjusting the way the business operates, as a precursor to looking at technology changes)
  • Application architecture (usually focusing on rationalisation, as a result of merger of unmanaged proliferation)
  • Service architecture (usually focusing on developing approaches for service governance)
  • Integration architecture (often now related to SOA initiatives, but in the past oriented around ESB, EAI initiatives).
  • Security architecture (unfortunately usually as an after thought).
  • Meta data and data architecture (unfortunately usually disconnected from the processes the data exists to support)
  • Rules architecture (usually oriented at understanding how strategies and policies are implemented in processes, procedures and systems)
  • Technology platform architecture (usually focused on getting better utilisation from servers, and/or planning for building extra server capacity)
  • Storage architecture (usually focused on preparing for the addition of extra storage capacity).
  • Standards management (usually oriented at governance)
  • Programme management (usually oriented at understanding how a set of capital initiatives, or business change initiatives relate to how the business seeks to operate in future).
  • Service level management (usually oriented at SLA templates, contracts and delivery exception management)
  • New product or channel definition (usually products that involve systems at the heart of their delivery)
  • Disaster recovery planning (usually leveraging off existing information about processes, applications, platforms and teams)
  • Business continuity planning (as for DR planning)
  • Requirements management (in the context of how the business seeks to operate)
  • Package implementation management (usually focusing on understanding how business operates, and how this relates to the package in question - so either the package can be changed or the business can change how it does things (or both).
  • Compliance management (understanding how regulatory or legislative compliance is being achieved)
  • IT Skills management (understanding what skills are required for what systems, usually to allow rationalisation or as a risk management exercise).
If one looks around an IT organisations one will see information about all of the above being created, collected, used and disposed of (whether that is intent or not). It is usually presented in a diverse set of office documents (business plans, business cases, project charters/statements of work, technology documents, software design documents and models, infrastructure design documents and models, contracts, requirements documents, business continuity planning documents). This suits each person producing the specific artefact admirably and keeps lots of staff and consultants employed. To change this situation take time and needs to be done incrementally and with purpose.

Often when you ask an organisation what they want to focus on they will say: applications, infrastructures (servers, storage, intregration) etc. but be unclear about how they will make intelligent decisions about these things without understanding: business/technology strategy, products/services, and the business processes, information, rules, organisations etc. that these solutions are designed to support. Or you will get organisations that say they want to focus on all the above - but struggle to work out what will the initial focus should be so that the stakeholders with greatest immediate need will be provided answers to the burning questions they have, while knowledge is gathered and organised in such a way that it is reusable in down stream phases to answer questions that are currently of secondary or tertiary importance (or not even thought of).

The art of a successful implementation is prioritising - based on a complete understanding of what the tooling can do, a good understanding of EA best practice and clear understanding of what the organisations current goals and challenges are.


What we sought in a tool for business/technology transformation and EA

(based on many people asking what the requirements are for a tool)

People often ask me what lead us to working with the tool sets we use for enterprise architecture and business transformation (EA/BT). The following is a short background.

Obviously the overall objective is to improve an organisation's effectiveness (reduce costs, improve income/outputs, ensure regulatory compliance, build asset value etc.). This requires managed change. I had focused how to improve the quality of transitions from RHE's inception (the best people, methods and tooling). We had tried using a wide range of OTS tools (e.g. CASE, Ontology, Document management and of course Office suites) and building tools to allow us to manage this information. After many years of trying to determine how EA/BT work could be done (in a way produce sustainable value for clients) and a long term focus on methods (including frameworks) we determined the tooling was inadequate (and the tool limitations fundamentally affected the method).

In 2002 we reassessed what tools were available that could be used to support:
  • business strategies: drivers, plans, markets, products/services, locations, organisations, resources, change/transitions, projects/initiatives;
  • business operations: communciations, services, processes, rules, information, organisation etc;
  • technology architecture: enterprise, solutions, components/systems, etc. Supporting any: styles, topology pattern, interaction models etc. And integrating detailed design models (e.g. UML, ER, BMPL/N, SOA) etc. and integrate data from other sources (e.g. CMDBs);
  • business and technology alignment: relationships and dependencies between the business and technology;
  • requirements (business/systems): at various levels of detail supporting an understanding of acceptance/contracts/terms and design/delivery;
  • change over time: time based views of all of the above (as-is, to-be, transitional);
  • change initiatives: programmes, projects (e.g. costs, risks, resources, timeframes) and business cases (e.g. benefits, fit to current environment etc.);
I wanted to ensure that knowledge in all areas (usually oriented at different audiences, for different purposes, but ripe with opportunities for reuse) could be inter-related i.e. so that information doesn't just reside in isolated silos (e.g. a plethora of office documents, or isolated models). I also wanted a focus on models that record explicitly the essential semantics (rather than just how pretty the pictures, are or how elegant the wording is, as often happens in free form documents and diagrams). This knowledge hub, with various interchange information mechanisms, would:
  • manage the semantics: model any object, and relate it to any other object; for a predefined library of object we would have a predefined set of properties, methods and relationships to other objects; it would be easy to change the modelling language (objects, relationships, properties, methods, appearance, nesting etc.); and we would be able to structure the knowledge to support any framework (taxonomy e.g. Zachman, FEAR, TOGAF, eTOM, etc.) so these could be tailored to specific customers/project needs etc.)
  • communicate: create visulations based on any sets of objects with the appearance, level of detail, layout, type information displayed, etc. changed to suit the audience/purpose; allow all authorised people to see the subset of the information they are interested in and annotate/revision-mark that information; allow publishing in any format e.g. office documents, images etc. and via customisable web interfaces (e.g. to support mashups etc.)
  • integrate: with other sources so we could exchange data with modelling tools; databases/applications; systems tools (e.g. CMDBs); portfolio management tools; programme management tools; dedicated modellers (e.g. project plan, UML, BPM, Data/ER, SW architecture analysis tools); etc.
  • easy to use and automatable: allow users to very quickly and efficient create models with very little knowledge of the tool (and very little training), to support workflows associated with maintenance and use of the knowledge.
  • open and extensible: to be standards compliant and implementation technology and technical architecture agnostic. To be able extensible so it can be customised to exactly what we or a client believes is needed.
Eventually (in 2003) we found tooling that provided a client visual modelling capability and a simple server side repository. During the initial implementations it became apparent that to be successful a combination of visual modelling, forms (web forms/dashboards) would be required to make implementations effective for large organisations (and to overcome organisational impedance). It was also recognised that essentially what the solutions represented was effectively a data warehouse /BI solution for the CIO (and IT) and that typically this was the one area of the enterprise that was not served well by IT. So in addition to the above what was needed was:
  • powerful analysis and reporting: to be able produce reports (using standard reporting tools) in whatever form (visual, textual, dashboards) and type (Word, PDF, web forms etc) is required and access these reports from portals and client tools so the knowledge can be visible to all stakeholders in a way to suits there specific needs, roles at each point of interaction.
  • tool to ensure the quality and currency of information: so we could automate the collection of data (where there is another source of record e.g. ERP, CMDB, etc.), to maintain policies on data (that allow data quality to be monitored/assessed), to allow easily ad-hoc updates by many users (who can do so as part of their day to day job) and proactively survey/poll users (when it is believed the data is out of date).
  • secure, scaleable and accessible: to support a large number of ad-hoc users (many that will only interact with the systems for a small % of the time but all of whom have an interest or are guardians of bits of the jigsaw puzzle.
  • tailored to specific purposes: solutions (based on tailored metamodels, user interfaces, system interfaces) to support specific jobs e.g. enterprise strategy or architecture; business cases and investment plans; technical architecture e.g. solution/integration/service/data/storage; or managing requirements, metadata, design, acceptance, programmes of work, security, business continuity
  • sophisticated management of multiple states: as in reality large complex enterprises have many initiatives operating each of which is expected to change the current state (at different points in time) and automate the way the changes will ripple through those states.
So in summary our current requirements could be summarised as:
  • suited to business (and technology) orientation - so ideally one could start with the business side e.g. able to manage business drivers and requirements and solutions.
  • flexible, extensible and customisable (language of modelling, user interfaces, data interfaces, datamarts, analysis and reporting) - so modelling can be done in terms/language suited to the business
  • powerful visual analysis and representations (for large amounts of data, power users, and design)
  • supporting a broad and diverse community of users (with suitable roles and security controls for those different groups of users)
  • automated data collection
  • communicating in various ways: diagrams, charting &  analytics, reporting etc.
  • multiple states and transitions between states
Having resolved the tooling issues – we then focused back methods – because it was clear that many initiatives in this area failed because people did not have a well developed implementation methodology. See (EA implementation)

Tuesday, April 22, 2008

Contemporary technology platform design

(shameless plagiarised from this work on SOA - real time enterprise)

IT Platform strategy needs to be driven by an organization’s business priorities. It is these priorities that set the drivers from which an application, system and network infrastructure strategy can be created. The increasing importance of IT as the digital value chain of the business, positions IT executives and their organizations to be more strategic and critical than ever before. IT organizations must transform from a “lights-on” culture to a “strategic differentiation through IT” culture.

An IT platform should:
  • combine decision support and just in time fulfillment with real time execution & information availability
  • deliver services when needed- as needed based on the explicit service requirements of the business.
  • afford organizations the ability to implement, new, unique products or differentiated capabilities in a timely manner for purposes of competitive advantage & operational control
  • leverage and reuse common component services (both business & infrastructure) for productivity, efficiency and competitive advantage purposes.
  • provide the necessary “plumbing” – rich user experience, dynamic execution environment, intelligent mediation platforms, real time data frameworks, optimal system footprints, dynamic network coordination, automated orchestration and tooling to enable autonomic management, monitoring & reporting
Some common questions driving a move to new platform are how to:
  • become more flexible and responsive to the dynamic needs of the business
  • simplify & reduce the complexity of the IT environment
  • get more value out of project & operational IT spend – both systems & people
  • reduce costs while delivering improved service
  • eliminate dedicated silos of data, systems and infrastructure as they exist today
  • reduce the time it takes to build and deploy new business services
  • implement and sustain predictable qualities of service
A platform has to produce services that respond to the business in four key areas:
  • business services – (information, automated logic, intelligent analysis)
  • infrastructure services – (federated query, caching, execution, messaging)
  • infrastructure resources – (storage, network, compute)
  • systems performance management – (service execution that alleviates/minimize any compute, memory, I/O or bandwidth limitations and meets service levels)
This helps address datacenter management challenges of:
  • Data Center optimisation: space, power, cooling
  • Network bandwidth optimisation and management
  • Enterprise systems management
  • Dynamic infrastructure management (operational processes and technical skills)
Consequently the key attributes sought in the design of IT platform are:

Service orientation
  • components and services are loosely coupled
  • components and services are not locked to an implementation technology
  • business logic and data is not be hard-wired to data stores or a technology/vendor
Real time infrastructure – a virtual, dynamic runtime environment
  • service requests are via: message framework (publish and subscribe), grid service (scheduled, on-demand, adhoc), fabric service (application server container originated)
  • services are dynamically allocated based on: service contract requirements of speed, throughput, load, calendar, wall clock, costs/margin rules etc.
  • performance management (applications, network) and dynamic load-balancing and message brokering
  • consumption and usage monitoring, logging (task, resource, user, transaction, job, etc.) and reporting (usage, service level compliance, trends, service allocation, efficiency, etc.)
  • policy driven infrastructure provisioning & configuration management
  • real time transaction workload management, transactional data caching and synchronization.
  • meta-data repository and data transformation services, meta-rule repository with real rule evaluation.
Service oriented infrastructure utility – policy driven consumption and fulfillment
  • ITIL/ITOM best practices
  • Proactive capacity planning
  • Multiple service fulfillment strategies
  • Dynamic network routing
Service oriented product management – repeatable discipline that defines the products, services, policies and infrastructure for service oriented business.
  • explicit management of the alignment of the business (strategies and operations) and IT - Service management that is suited to the granularity of services provided and consumed (service level management, service life cycles, service catalogues)
  • systems integration architecture and team
  • utility infrastructure model oriented at end to end optimization
  • asset management (for reuse).