The Waterbed Theory — Complexity’s Devious Compensation Mechanism | by Guy Erez | Oct, 2022

Tips to keep your software engineering features balanced

Photo by Linus Nylund on Unsplash

Have you ever heard of the Waterbed Theory? I’m assuming you’re all familiar with waterbeds. Waterbeds may be quite amusing, but the main issue (at least with the inexpensive models) is their stability.

If you sit on one side of it, the other side gloriously rises up to compensate. You might then try to balance that part out, only to watch other parts rise in its stead. That makes balance nearly impossible to come by unless you hold completely still. That doesn’t tend to be terribly convenient, which is probably why waterbeds have gone out of style.

I’m not here to discuss waterbeds though, as amusing as they may be. I want us to use this bumpy experience as an analogy for other areas in life, and in our case — software engineering.

Good software engineering is all about managing complexity. You don’t want your code to be too complex; otherwise, it’ll be nearly impossible to maintain it. You don’t want your apps to be too complex; otherwise, your users won’t be able to enjoy them. Complexity is fine when you’re its producer, but when you’re the consumer, things tend to be quite different. That’s why a junior engineer might revel in their complex solution to a problem while the senior engineer will think of its consequences — unmaintainable code.

Photo by Brett Jordan on Unsplash

There’s just one problem. Managing complexity in software is no easy task. The Original Waterbed Theory discusses this exact problem. When you try to reduce complexity in your application, it tends to simply “shift” from one place to another. Let me illustrate this issue with two real-life examples that taught me this lesson the hard way. They will help you understand the tradeoffs we implicitly make when building software solutions.

One of the more common examples of unintended complexity is the All-in-One SaaS application. On the one hand, it’s a one-stop shop for its customers — all of their needs can be met using just one application. On the other hand, nobody can actually use it without spending thousands of dollars on “academy courses,” dutifully provided by the SaaS application’s company. That’s a pretty decent business model for the SaaS company, but it doesn’t really provide a great experience for its customers.

Its proposed solution introduces a shift in complexity, not a true reduction. Yes, you don’t have to use 2–3 different apps to satisfy your requirements, but your all-in-one app requires such a steep learning curve that it’s practically unusable. Now don’t get me wrong, I’m not saying that every feature-rich SaaS application suffers from this predicament, but from my personal experience, that is quite a common issue that still produces challenges for my team using such an app.

That’s another interesting use case I was personally involved in. We wanted to build a robust data pipeline that will be super easy to maintain for developers, and even easier to use, as it was meant for non-technical users. To do that, we’d decided to build a low-code solution, heavy with configuration.

Our assumption was that having extremely generic code would enable non-technical users to easily configure new marketing events on their own. No more waiting for the dev team (that’s us) to help them. A win-win situation, or so we’ve thought. To make that a reality, we needed to eventually connect the backend to some neat UI. Time went by, and we couldn’t find resources to do that. Moreover, it felt like even if we did build a UI, it would still be quite difficult for non-technical users to use it, as the pipeline became more complex than we expected.

That’s not too bad, we thought. One of the reasons for the project was to improve our team’s velocity, so it’s quite alright if we’ll be the main consumers. And indeed, the process of supporting new marketing events has improved considerably. Most new events require a simple configuration setup, and it’s good to go. However, we found more and more events to require non-trivial changes that would have been much easier to do with simple generic code.

The configuration ended up reducing complexity on one side — setting up generic events via configuration while increasing complexity on another side — supporting non-trivial events. Also, the ratio of trivial to non-trivial events was less favorable than we assumed, so it further reduced the aforementioned benefits.

To be honest, you can’t always anticipate whether a project will be a “waterbed” project. It’s hard to anticipate how reductions in complexity might cause complexity to rise in other parts of the system. As we experienced in our data pipeline project, our expected ratio of trivial to non-trivial events was off by quite a bit. That kind of data is hard to come by until you’ve already implemented the solution.

The main thing I found useful was to start small and focus on simplicity. A simple, working solution, is often better than a complex solution that offers more features, as those tend to be harder to maintain and use. When you want to add more complexity, always think about the tradeoff, and whether you’re actually going to reduce the overall project complexity.

Oftentimes, you’ll simply shift complexity from one part of the system to another. Focus on solutions that require a one-time complexity investment, that pays simplicity dividends once it’s complete.

Also, try to find people that had experience implementing similar projects. They’ll often be able to steer you away from “waterbed” solutions.

The Waterbed theory, like its physical counterpart, is quite an interesting piece. Its main thesis is that many projects don’t truly solve problems, they simply shift complexity from one place to another. Our goal is to actually solve the problem, or at the very least, reduce complexity holistically.

To do that, we need to be aware of the second-order effects of our solutions and focus on simplicity. Keeping it simple might not always produce a perfect solution, but it’s a surefire way to reduce complexity, which increases your chances of producing a useful solution.

News Credit

%d bloggers like this: