Technology transformation -key learnings..
Context Setting
Having lead technology, Fintech and architectural transformation endeavours I thought posting this blog to share my perspectives…
The approach may differ when driving transformation in a large tech organisation vs. smaller technology company but I don’t think the fundamentals are different. From my point of view there are three fundamental dimensions to look at to be successful;
Maturity and state of the technology and functional architecture is and what “good looks like” from business, process and technology architecture perspective, both short and long term creating a sustainable delivery roadmap
Maturity and the quality of the engineering and technology delivery processes, tools and the rigour
The overall culture to embrace and adopt change as well as the organisational set up to facilitate the change
Three dimensions needed to achieve successful technology transformation
I was fortunate enough to drive all three dimensions in recent engagements and what I learnt is the critical inter-play between all these to make the overall transformation a success. Also, I have learnt from past mistakes that putting too much emphasis on the architecture instead of delivery will result in not delivering the right solutions in time as well as focusing more on the first two topics and not on latter resulting in poor sustainability.
The transformation requirement may materialise due to many reasons;
- Cost and operational efficiencies – the need to reduce duplicity, inefficiencies, redundancy of business functions, processes and technology landscape
- Market differentiation and need to reposition: Like many big financial giants facing these days , how to be nimble, efficient both from cost and delivery perspective to meet the challenge from smaller challengers / FinTech institutions
Both needs have their own complexities like;
Do we have short term ( yet sustainable ) options to achieve efficiencies ( instead of pure cost cutting or offshoring)
Can we create a pragmatic incremental delivery plan and an architecture so that value can be delivered (and proven ) sooner than later? I have seen many programs with multi-year ambitious plan with great “architecture on paper” yet lacks strong technical design and delivery to prove and then fail after few years.
Software engineering vs. other engineer practices
Other engineering practices like Civil engineering has an illustrious history going to thousands of years. Mechanical and electrical engineering also has 100+ years of maturity and learnings. Whereas software engineering is still in its infancy in my view. The fundamental general computer architecture is still based on von-Neumann architecture.
The cost of change in other engineering practices is quite high. Imagine half way building a sky scraper, you realise a massive design flaw and then have to refactor the architecture. But even in large software delivery programs I have seen companies draining 100s of millions of dollars in multi-year projects without actually realising successful delivery.
While on one dimension we can argue that the correction cost at the end can be the same, the approach to making corrective actions in Software projects can be much easier.
for e.g.
Software is malleable and ability to change quickly is higher compared to say a civil engineering project.
Introducing change to composability and modularity of a software architecture can be easier compared to changing the interfacing in a physical engineering product. Yet a poorly architected and delivered software product may take longer to deliver compared to delivering a mechanical engineering product that has clear modular design and mature engineering processes
The rate of change in requirements for software project is much higher compared to a massive construction project. Requirements and tools can change in very short time frames like few months. Having said that other industries like Mechanism engineering, especially Formula 1 , the ability to change the designs and parts quickly like even in a single season has played key to some recent F1 Manufacturer winners like Mercedes.
With the use or Test Driven Development (TDD) and Behavioural Driven Development (BDD) , velocity in the testability in Software products has increased rapidly. Adoption of Continuous integration, Continuous Deployment and Continuous Delivery has further improved the ability quickly test, change , verify and test again making the “ability to change” faster.
In this first article in Technology transformation I will be focusing on the first topic, Architecture transformation…
Architecture transformation
First let me point out some practices I have adopted in in the past which has worked better than some of the other less successful practices..
Formulate a high-level view of the strategic state
When I started my software engineering career, we used to deliver in Waterfall methodology. The overall application and systems architecture as well as the detailed front to back inter-system architecture was formed up front. While this gave clarity to every one, it was hard to change when new requirements were found later or when a need for the overall front to back process was further refined. The users only saw and interacted with the systems quite late that always resulted in changing the requirements resulting delays.
Hence what I have done recently is formulate ONLY a high level functional, data and solution architecture instead of a very detailed architecture. Starting from a high level functional domain, data domain, high level APIs of each domain had been very helpful in subsequently formulating a evolutionary and detailed solution while developing the solution;
Front to back business and functional architecture
What functions needs to be built to achieve your long-term business goals. What key differentiators need to be built to achieve the business goals. Create functional blocks and what each function does. This can be from pure business perspective like user features and epics that define the shape of the overall requirements
Front to back data and data integration architecture
Coming down from functional architecture , map what data each component own/masters as well as what other information functions need from other functions. This is where I have started adopting a domain driven architecture modelling to get;
Clear functional responsibility for a service or set of services
Clear data ownership ( what entities that the domain is responsible for)
And how these functionality will be exposed and how these data will be accessed ( through clearly defined APIs) in a consistent, atomic way
Solution and technical architecture
When a good understanding of the high level functional, data and access requirements are formulated , it’s a good time to look at the shape of the solution and technical architecture of the overall components. Surely a significant focus needs to be given to non-functional requirements like performance, scale, usability/access requirements (web, different devices, user base and coverage), security profiles and needs , availability, latency, data processing needs, data foot print / growth needs as the solution and technical architecture WILL BE BASED ON THESE. For e.g. if I am building a real time surveillance system for cash payments, key non-functional requirements include 1) ability to process request with low latency 2) Ability to make decisions based on different profiles and monitoring dimensions quickly 2) Efficient and compact data structures to hold different dimensions of surveillance (temporal, numeric, location etc) etc
Hence I have always spent good time up front to get a view of the shape of the solution and possible technical options ( HL) to implement the solutions
And also itemise any critical technical items that needs up front assessment or PoC’ing that can be proven fast / or failed fast( discounted) so that alternate solutions can be looked as sooner than later.
User and process architecture
Another dimension is formulating the end user process and engagement architecture. This can be UI/UX or any other ways of users interact with the system using any mode of device. Using Low code/ No code tech , it has been much easier to deliver end user and behavioural experience up front to the users. Again focus should be to
Get key usability features understood and exhibited
Test the business , operational and cross-user interaction processes up front, refine and refactor with low investment
While understanding the key components (technical and functional) that needs to be built in the actual solution
Work on an evolutionary architecture
Post formulating the HL architecture, it’s essential to have an open mind that “the detailed architecture is not cast in stone” and it will evolve.
The overall shape of the architecture can remain same but the details of internals ( business logic implementation , data models, API models) will change
And some times the shape of parts of the architecture can change. This is really important and this is where the need to
Identify high risk features/epics that need technical and solution PoC: it can be highly technical as PoC’ing out a low latency event processing architecture that need non blocking concurrency/IO with close proximity to high performant memory VS very user focused proof of a complex trading interface.
As a technologist I always try to find what these “high risk” features are and then formulate PoC/ pilot’ing these features and testing the waters, in early sprints
But the point I really like to stress out that its ESSENTIAL to;
Get a good confidence of the solution architecture up front. Spend initial sprints in refining the hot areas of the architecture.
keep a GOOD BALANCE between delivering user features and realising the solution architecture. I have seen some times that the development team only focus on the end user functional without looking at high risk technical items. And at the end when those features that actually stressed the architecture’s limit, it required a re-write some of the key components of the architecture.
When delivery scope is planned, ensure both functional and non-functional features are prioritised. for e.g. when I was building a real time credit risk monitoring solution for high volume trading business, the immediate high risk technical items came to mind were 1) the ability to execute and evaluate rules real time 2) the dimensionality of the rules - temporal, sliding window, cross entity/ the hierarchical nature, aggregation/composition, statistical deviations 3) concurrency and ability to parallelise for different accounts/counterparties 4) diversity and volume of data needs to be stored at different layers ( in memory/ disk) based on the need). Here I managed to formulate technical stories that were prioritised at the beginning to select the correct technical architecture.
..But not an ever changing fragile architecture
One of the comments I some times hear from developers is that “ we will figure that out in the next sprint”.. While it will work for non-complex features ( adding a form to a UI to edit a trade) , some times key technical items need to be formulated up front. I was asked to step into salvage a massive program where a ledger system was getting trades, risks, positions, prices from 1000s of systems. The team had adopted a “Cool architecture” at that time - Hadoop and HDFS without actually looking at the required atomicity, concurrency, the need for real time and batch based processing. While the decision to select Hadoop was not clear and explained by the team ( it was big data tech!!) , still the solution would have worked if the actual usage of Hadoop and the user/data/integration scenarios were identified as critical items and solution vetted with a PoC. I spent time refining the architecture and guiding the delivery team to refactor the solution..
It’s critical to test the architecture repeatedly and proactively with critical use cases and scenarios.
Start with a walking skeleton
This term comes from evolutionary architecture and agile practices. What this means is don’t build the full solution, try working on a small vertical slice of the problem to realise the solution and then adding “flesh”. But I really want to highlight few key points. if you are looking at a skeleton of a homosapiens, then focus on the architecture of that skeleton and what it has to do … and not focus on a building a skeleton of an elephant ( each skeleton is different from each other .. the characteristics , behaviours and expectations will be different, what they can/cant do is important to know!).
Hence select key behavioural, usability, functional and non-functional characteristics to build the initial skeleton (and not ancillary features)
Then build based on that skeletal after you are very confident that the skeleton ( hence the structure, connectivity ) will serve the purpose , start adding “meat to the bone” - i.e. start adding the functional and non-functional features.
Make sure the walking skeleton forms CRITICAL architecture considerations for the system. i.e. say you are designing a portfolio optimisation tool for an asset manager. This requires looking at various pricing curves, instruments, optimisation constraints and scenarios across large cross-cutting data sets which requires large data processing. Hence selecting a distributed / parallel data processing / storage/ execution and scheduling architecture is a key consideration needing 1) scenario simulation 2) what-if and shocking.
PoC out these key considerations first
Write down these key features and formulate architecture stories / PoC / Pilot stories.
Start selecting few architectures and tools that would perform well in the key architecture stories
Tactical v. Strategic solution formulation
I intend to write another article on my experience ( good and bad) in Tactical vs. Strategic engineering but its essential to balance both. While tactical architecture will show immediate benefits and perceived speed in delivery, if time and time the solution had to be changed to accommodate the strategic goals , the throw away and cost of refactoring will be high making the overall project a failure. Therefore as mentioned above;
Try to spend time initially to get the key architecture and technical skeletal done to ensure the key features and considerations are reflected in your design thinking
Then start formulating the step by step feature build based on the prioritisation
Use tactical builds where there is true immediate value like creating working UI/ UX to get users feedback sooner ( mocks with simulated back ends). Use low code/ no code tools to reduce the time to market to showcase the usability and functionality, leverage the low code tools to tear down and build new functionality to ensure end user understanding is solid before you delve on actual engineering.
Also spend time in assessing short term vs. strategic changes
In one of my previous articles I talked about how tactical /short term as well as architectural/strategic transformations can be delivered to realise solutions. Its vital especially in a large transformation program to look at short term solutions bring some “breathing space” until the more strategic transformation can be brought in.
I have adopted short term remediations in many transformation projects not just to bring immediate benefits but ALSO
To understand the problem better like;
Most of the time the problem perceived by the user is not the actual problem. Some times the solution is a simple business process change or using simple tools instead of building a new application.
To further understand different dimensions and levers to the problem ( Is that a usability issue, non-functional issue? ) which will surface “quick fixes”
Talk to the team on the ground and get feedback from various parties. some times the people performing the strategic solutions don’t really know the problems on the ground
Ensure the tactical solutions are time boxed and constrained:
I have seen many times that more focus is given in building tactical solutions to resemble strategic solution. But that is not the objective of short term solutions. they should cost much less, take much less time, and above all less risky. Ensure the fix has limited horizon to ensure the strategic solution is also delivered (and not forgotten).
In my next article I will focus on the other dimension , engineering delivery and creating delivery roadmaps , focusing on fail fast, agile, automation tooling
