On Architectural Complexity

  

  Complexity is one of the major problems plaguing software projects today. When the architecture is frozen and they have run the product, the number of features grows with the size of the code. This rate of growth is seldom constant.

  It is a shame that customers perceive features in complexity when they are meant to be seen as architectural. It has been my personal experience that while the number of product features increases logarithmically, the development team size increases sub-linearly and its engineering team size increases linearly. The time to implement these features increases exponentially.

  Coding is an act of fighting complexity. It rewards to study what kinds of code modifications increase project complexity. I equate the complexity of the project to the complexity of its most complex component. However, the reductionist approach to identify that one component may be complex by itself.

  Each line of code has its say in the highest conceptual level of a program. The topmost layer of the program conceptualizes various components lying on lower levels of independent modules, and so on until we reach each line of code. Understanding these pathways from the bottom levels to the top is termed architectural awareness. If the coder has none, the program will blow up fast.

It is important to acknowledge that requirements from the customer evolve to the eventual coder. This evolution happens on three fronts:
  1. Requirements from the customer: stuff that defines the product.
  2. Requirements from the architect: software model to be implemented that is proven to mimic customer's definition of the product. This basically sets the interaction protocol among various components.
  3. Requirements from code: functionality and test requirements.

  Failure to treat the specification under these three requirement headers leads to ad hoc style programming. Projects thus handled are difficult to manage.

  Any software development process can leverage the extent of the design of its final product, but cannot define it. If a product reaches its complexity limit, it becomes impossible to carve out a reasonable development process out of it. It usually reaches this limit when new stuff breaks old functionality.

Some stuff I learned over my years of good and bad design decisions:

  • When two designers meet, their stupidity adds up. Not their intellect.

  • We are not smarter than our circumstances. Just sometimes, our circumstances do not understand us.

  • Intel's instruction set architecture is so complex that intelligent design must have devised it over the years. But then, it is so complex that no intelligent person would dare to engineer it.

  • 90% of stuff around us is Dark: Dark matter in physics, Dark non-coding DNA in biology, Dark money by contracts in economics, Dark information of inconceivable events, and Dark effort estimations of software projects.

  • The laws of software bugs are simple but too weird to discover.

  • We must copy things that can be copied. Life and science have taught us at least that much. One copies its aspiration of gaining its presence, while the other its aspiration of gaining knowledge.

  • We should never ask people to do a job. They bear subconsciously in their mind that it is our responsibility to take care of that task and are less likely to take an initiative in the future.

  • Why is documentation important? Because it freezes information in time. Otherwise, this information that has sense will only be included in the future, and different senses can cause different interpretations.

  • Every beautiful product starts with engineering and ends with it. If engineers talk about something new, it is a sign that a new design is to be born. If they talk about a particular product, be sure it will be dead soon.

  • Shared storage of duplicate files will come soon. But not too soon.

  • Every software project, big or small, comprises simple and elegant modules. It is their mosaic that brings about its complexity.

  • Simplicity pays for functionality. Spend wisely.

  • It is YOUR job to see new. Seeing an analogy or a composition of well-understood things when seeing something unusual is not why you are being paid for.

  • Intelligent design is the opposite of repetition.

  • In AGI, everyone is a sage.

  • Understanding = Programming.

  • Wonderful products are always the result of either a simple algorithm or an aggressive elimination of cruft.

  • Small programs require only formal thinking. Large programs require nothing but art.

  • Security of the system you build is only as strong as its greatest incentive challenging to break it.