Why Is Software Development Expensive?

Why is software so expensive? The software inflation rate in 2020 is 2.2%. It is getting ever more expensive. Once you get past an MVP, you suddenly end up with a test team, a project manager, a marketing/finance team, or investors. Things start to not go as planned all of a sudden. Why do development costs increase? Is it just a scaling thing? Is it the user expectations? Just like movies go over budget and time to match expectations, are software businesses managing to make things complicated to overrun the project schedule?

If a software company decides to outsource a small job, it can suddenly end up with extra people, becoming an overhead. It seemed a cheap option at the time, but now it doesn't seem so that way. It buys a tool to fix some of its problems, but the limitations of the product are not honoured and the company ends up buying another one, then another, and spends much more on integration costs. We have infrastructure costs, licensing costs, utilities, security and other management costs to add to.

With hundreds of pages of requirements, numerous edge cases, complicated pipelines and fail processes, there is a complex interdependence between a host of systems. Stuff is getting complex. Complexity is the mother of creativity. There is complexity in understanding, in designing, in maintaining and in selling. The price of employment is increasing rapidly because there are ever more developers competing on salary.

It is becoming very difficult to retain large teams of career programmers who have worked on the same systems and technologies within the same industry for 5+ years, supported by business personnel with same experience. This makes any software company susceptible to expert consultants, snake oil marketing and technological fads. Cost-efficient software is not being mass produced anymore. Venture capitalists are starting to believe that the costs are increasing when infact it is the coder's cost-inefficient design knowledge that causes the problem.

This phenomenon has reached its critical mass in that we have enough programmers who develop cost-inefficient software with unrealistic prices, making this a new normal. This discounting the fact that cutting-edge tech is always expensive. Though the cost per unit utility is decreasing, custom made software is getting ridiculously pricey. The technology stack used to be very small and stable. A career programmer could have come to grips with the technology using about four 300-page books. The tech stack today is exponentially larger and more volatile. The tools to plumb together a working artefact are readily available, the support, scale-up (or scale-down), process optimization costs are on the rise.

There is a growing demand for developers, and we are yet to reach its demand peak. The supply of developers is constrained by capacity/desire to perform tedious tasks of learning development skills. Very few companies allot time to refactor their code to make it more reliable. Most of them are on the "move fast, break things" mode. The direction is more towards optimizing towards shipping speed. The amount of rigor and research that goes into a software system depends wholly on the business requirements. But these aspects are conveniently ignored by all parties somehow. Building a prototype to get to the market fast and subsequently build on top of that prototype makes it harder to bring changes to the system and have it behave correctly. It creates havoc on many fronts.

Many managers discourage thinking and coerce their subordinates to produce code fast. They then complain that 90% of their human resource is tied to code maintenance, and blame the tech stack for such a sorry state of affairs. All of this is well known; it just pays to be said again here. All such inner complexity is not something that the client cares about. They are unsure of their requirements mostly; want a feature; and want it right now. As the dev team inexorably grows to several times its original size, good developers responsible for a majority of the codebase move on to another company and the number of features are reduced till a point is reached where everybody is doing maintenance and features start to break.

Then comes in our infamous ticket racket from our CTO where teams compete to close various irrelevant tickets. Teams surprisingly work overtime when they are in this mode. CTO shockingly discovers a rekindled spirit among developers, registers their dedication and work commitment and the CEO is impressed. Then comes in a project manager "to manage" this exponential blow-up of tickets. The company manages to snag a dedicated guy willing to deal with this mess for 3x standard salary.

I see this phenomenon everywhere. Not everybody seems to appreciate the benefits of not adding unnecessary features and incessant changes that complicate a system, without adding much value. In fact, 90% of the features are never used 90% of the time. With zero technical oversight, any outsourcing company welcomes such unhealthy developments as this means more billable hours for them.

The solution to wade through this unwelcoming mess is through elegant mathematical formulations and a terse, simple and tacit technical sub-stack. For the lack of appropriate theory in workflow and process management, problems are solved with common sense. If code is written keeping a thinker in mind, costs would drastically come down to a fraction. Code when written for the next average guy to understand and maintain, begets complexity. Any business with 3-4 thinkers is orders of magnitude efficient, productive and profitable compared to businesses having armies of average programmers optimizing for code maintainence.