Improving Business Agility and Providing Continuous Value | Insights

Agile development and DevOps are now standard practices for insourced development teams at leading global companies and among many startups. In contrast, the waterfall model remains prevalent and misconceptions about Agile development are widespread in Japan, leading to persistent challenges in improving software quality and delivering continuous customer value. This insight explores the root causes of these challenges and vicious cycles faced by Japanese IT, how organizational reform and cultural transformation address these challenges, and how companies can establish sustainable software product quality.

Kazumi Moriyama

Senior Manager

The legacy of the “Three Lost Decades”

Japan, once the most internationally competitive country and the world second largest economic power in the world, entered a prolonged period of stagnation in the early 1990s. Having experienced the “Three Lost Decades,” Japan is steadily on course to extend this to a fourth decade. The Wall Street Journal noted in 2020 that the these Lost Decades were the result of Japan’s failure to implement structural reforms. While direct foreign investment increased steadily for ten years through 2023, growing about 4% YoY ^*1, the revenues from these investments have largely been reinvested overseas, meaning that crucial innovation efforts needed for domestic economic recovery have been overlooked.
Typical Japanese companies are organized as vertically separated functional organizations, where operational and IT departments function independently. Product development is approached as a one-off project, organized around temporary teams composed of members selected from various vertical departments. Projects follow the Waterfall model, with teams disbanded upon completion. This makes it difficult to deliver value to customers in a timely manner. Processes and knowledge gained during projects are not retained, future-oriented continuous improvement does not occur, and no organizational culture is fostered.
In software development, cost reduction has become the primary goal, depriving companies of internal development capabilities. Consequently, they frequently must outsource entire projects to vendors or defer to vendor judgement. This vendor-dependent model represents a negative legacy of the Three Lost Decades. The cost-cutting priorities have created shortages of talent needed for requirements development and product development, slowing companies responsiveness to change requests.
This reduced responsiveness has degraded software quality through End-User Computing (EUC) disorder, increasing operational complexity due to ad-hoc workarounds, and the proliferation of duplicated codes for short-term development. This accumulation of issues has created systems that no one can properly comprehend. Problem-solving consequently requires excessive time, further delaying response times and creating a negative spiral that consumes more resources and inflated costs. This leads management to conclude that IT costs balloon with insufficient return on investment, prompting further cuts to maintenance budgets. This represents the fundamental problem and vicious cycle confronting Japanese IT today (see Figure 1).

*1 Japan External Trade Organization (JETRO), Global and Japanese Direct Investment

Transforming organizational structures - What is a product organization?

So what should companies do to address this fundamental problem and escape this vicious cycle? Firstly, they need to revise their organizational structures. Organizational structures can be divided into three general varieties. These are “functional organizations (organizations based on occupation),” which are separated vertically as per the aforementioned departments, “project organizations,” which bring together people with expertise on a per project basis, and “product organizations,” which provide value originating from the customer throughout the product lifecycle (see Figure 2). Many Japanese companies tend to be made up of functional organizations, representing their legacy organizational structure, and project organizations. To keep up with the advance of digital technology and continue growing, however, companies need to transform into product organizations that enable the provision of value originating from the customer. Such transformations free companies from the cycle of forming teams per project then breaking them up, allowing them to continuously improve processes in a single team while building up a store of processes and knowledge created in the process of product development. This marks a product team, and an organization made up of multiple such product teams is a product organization.

For many Japanese companies, IT is just a tool for simply making their business more efficient, and is something that is unlikely to change often. This thinking has made it common for such companies to keep using IT systems once made without any significant upgrades. We have, however, arrived at a point where, across business activities, companies cannot survive without making continuous progress in their services levels and systems by continuously incorporating the power of digital technology. This means that the progress of digital technology is core to the business activities of companies themselves, rather than a simple tool, making it a necessity for them to incorporate that progress into their business activities. This means that this is not just “something for leading global technology companies to incorporate,” but rather that, regardless of industry, all companies have no option but to incorporate digital technology into their business activities. To incorporate digital technology in business activities, as a minimum, each business domain has to have a role that has responsibility for the product vision and value, and optimizes business KPI; a role that has responsibility for executing product delivery; a role that has responsibility for product development and operations (i.e., operations and maintenance) that continue to provide value to the customer, and that optimizes systems KPI; and a role that actually does the development work for the product.
Figure 3 shows the core team members and key roles that make up a product team, which makes up the smallest component element of a product organization. (Depending on the nature of the product, a product team may also include user experience (UX) designers or QA quality assurance (QA) engineers).

The team members who take on these roles need to drive product development as a single team. To that end, it is important to genuinely commit to changing attitudes to transform organizations in a way that genuinely fits with advancing digital technology in the direction of crossing departmental boundaries such as those between business and IT departments, and seeing members compromise with and learn from one another.

Organizational culture and mindset

With that said, for the typical Japanese company, there is one more barrier that needs to be overcome. This is the separation between development and operations.
In leading global companies, development teams typically take on the operations of the systems they develop, and a mindset that puts operational excellence (OE) at the forefront permeates the workforce. That is to say, such companies practice continuous improvement to maximize efficiency, quality and customer satisfaction across the organization, and constantly provide value to customers, while advancing business activities including systems KPIs, and not just business KGIs and KPIs.
In the IT departments of typical Japanese companies, however, development and operations are made up of separate teams, and coordination between the two is lacking. Development teams are made up of a single or multiple vendors, so having established team members is difficult. It is also rare to see knowledge sharing, the establishment of operational processes and improvement activities take place within such teams. This makes it difficult to foster shared understanding of the goals and value of the business or product, let alone a team culture. As a result, team members tend to limit their goals to fulfilling their assigned tasks, so it is often a struggle to maintain team motivation.
To address these issues, companies of course need to build environments using the right Agile product development methodologies and tools. But the most important element is to have a culture that permeates throughout teams, organizations and the entire companies. Lacking such a culture, organizations will not be able to exhibit high performance. American sociologist Ron Westrum put forward three models of company culture (“A Typology of Organisational Cultures,” 2004):

1)

Pathological (power oriented) organizations: Characterized by highly significant amount of fear and threats. Information is often hidden for political reasons or skewed for the sake of appearances.
2)

Bureaucratic organizations (rule oriented): Each department is protected. People preserve their own “territory” and assert their own rules.
3)

Generative (performance oriented) organizations : Focused on mission. People prioritize seeking how to achieve aims or achieve exceptional performance.

Generative organizations are the model used in DevOps, so it is important for companies to optimize their cultures to make their product organizations follow the generative organization model. For example, such organizations build frameworks for viewing failures as learning opportunities and then putting the lessons learned into practice. Leadership encourages constant learning and improvement, and provides an environment that lets people take the challenge of improving. Once mistakes are discovered, it is easy to go back to them, even if the decision is thought to be important and have a major impact on the future. Finally, culture affects software delivery and organizational performance, and leads to improved job satisfaction. The key elements underpinning such cultures are improvement and continuous delivery.

Continuous improvement for ensuring quality

A concept of improvement originating from Japan, known as kaizen, has been broadly taken up across the world. Through such kaizen, the Agile development methodology has progressed globally for over 20 years, and product teams optimized for Agile product development have become commonplace. In the process of this development, the idea of DevOps arose. This process of kaizen has continued on to this day, with Agile development and DevOps initiatives underpinning the continuous growth of companies.
But in Japan, where kaizen first arose, DevOps is yet to spread. Let us take a look at the main reasons why DevOps has failed to spread in Japan, beyond issues of organizational structure.
Some of the main reasons why DevOps has failed to spread in Japan, beyond issues of organizational structure, are: 1) the misunderstanding that Agile development requires abbreviating processes, 2) a lack of understanding around the need to complete exacting tests before commits and merge, and 3) insufficient application of development approaches that perform daily builds and tests.
Such misapprehensions have led to a widespread inability to ensure general software quality, due to a focus on delivery speed in underprepared environments. The separation between development and operations further runs the risk of preventing the lessons learned from problems in production leading to improvements in architecture or improvements to customer value, and thus, by extension, of developing into quality issues.
So, how can companies ensure high quality? To begin with, companies need to analyze their current situations and gain a proper understanding of Agile development and the DevOps approach. They can then build the right organizational structure systems for themselves, and ensure product quality as a result of practicing continuous improvement through product organizations. Quality does not mean making a plan, carrying it out and finishing things there. Quality is a living thing that is developed while optimizing systems in accordance with the growth stages of teams, organizations and companies as a whole. Let us take a look at a few specific approaches to ensuring quality. These would be policies including establishing clear quality standards, continuously improving metrics, enhancing testing quality, loosely coupled architectures and customer feedback loops.
Product teams perform product development through the dual efforts of Agile development and DevOps, and optimize their culture while continuously improving their systems. As a result, organizations achieve business goals in a stable and developmental manner, by putting high quality into practice as a matter of course, thus managing to transform themselves into organizations that generate innovation.

Summary

For Japanese companies to accelerate future growth, business and IT departments must collaborate to prioritize customer value generation and drive performance improvements in business operations and digital technology utilization. To put this into practice, it is important for companies to build organizational structures that continue to execute digital transformation quickly and with high quality, to put in place culture and systems that practice continuous improvement, and that drive transformation with a view towards overall optimization. Japanese companies can achieve high quality by having each individual once again take ownership of and practice the kaizen, exported from Japan, as a Japanese form of “improvement,” thus unfalteringly advancing such an overall flow.
ABeam Consulting is ready to contribute to improving the business agility and continuous value provision of your company, based on our strong track record of developing business and IT strategies for companies, developing new businesses and transforming existing businesses using digital technology, training digital transformation-ready personnel, and implementing and consolidating high-level Agile, DevOps and quality assurance approaches.

Insights TOP