April 8, 2020

Advantages Of Data Oriented Programming

One of the biggest advantages I've found working with Clojure is its data oriented nature. Ultimately, all that code is doing is transform data. A program starts with one piece of data as the input and produces another as its output. Mainstream languages attempt to abstract over that using object oriented semantics. While practical value of such abstraction is not entirely clear, there are some tangible problems associated with this approach. Let's take a look at some drawbacks to structuring programs using OO style.

Traditionally, an object can be thought of as a type of a state machine that contains some data fields representing its internal state and provides some methods for manipulating it. An object represents the smallest compositional unit in OO, and a program is structured as a graphs of such objects that interact with one another by manipulating each others state.

The first problem we have is that each object is an ad hoc DSL. When a developer designs an object they define its API in form of methods and come up with the behaviors the object will have. This makes each object unique, and knowing how one object behaves tells you nothing regarding how the next object might behave. Rich Hickey illustrates this point in detail in his Clojure, Made Simple talk. The more objects you define the more behaviors you have to keep in your head. Thus, cognitive overhead grows proportionally with the size of the program.

Any mutable objects present in the program require the developer to know the state of the objects in order to know how the program will behave. A program that is structured as a graph of interdependent state machines quickly becomes impossible to reason about. The problem stems from objects being implicitly connected via references to each other resulting in shared mutable state. This leads to lack or referential transparency and makes it impossible to do local reasoning about the code. In order to tell what a piece of code is doing you also have to track down all the code that shares references with the code you're reading.

This is one reason why sophisticated debugging tools are needed to work with code effectively in object oriented languages. The only way to tell what's happening in a large program is to run it in a debugger, try to put it in a particular state and then inspect it. Unfortunately, this approach is just a heuristic since there may be many different paths that get you to a particular state, and it's impossible to guarantee that you've covered them all.

Another notable problem with objects is that there is no standard way for serializing them creating additional pain at program boundaries. For example, we can't just take an object graph on from a web server and send it to the client. We must write custom serializers for every object adding complexity and boilerplate to our programs. A related problem occurs when composing libraries that define their own classes leading to prevalence of wrapper and adapter patterns.

All these problems disappear in a data oriented language like Clojure. Modern FP style embraces the fact that programs can be viewed as data transformation pipelines where input data is passed through a series of pure functions to transform it into desired output. Such functions can be reasoned about in isolation without having to consider the rest of the program. Plain data doesn't have any hidden behaviors or state that you have to worry about. Immutable data is both transparent and inert while objects are opaque and stateful.

As a concrete example, Pedestal HTTP server has around 18,000 lines of code, and 96% of it is pure functions. All the IO and side effects are encapsulated in the remaining 4% of the code. This has been a common scenario for the vast majority of Clojure programs I've worked on.

Cognitive overhead associated with reasoning about code is localized as opposed to being directly influenced by the size of the application as often happens with OO. Each function can be thought of as an small individual program, and we simply pipe these programs together to solve bigger problems. Incidentally, this is the exact same approach as advocated by Ken Thompson in Unix philosophy.

Data can also be passed across program boundaries since it's directly serializable. A Clojure web server can send its output directly to the client, and client code can operate on this data without any additional ceremony. I've discussed some of the practical benefits that stem from having standard serialization semantics in this presentation.

Another advantage of separating data from logic is code reuse. A pure function that transforms one piece of data into another can be used in any context. A common set of functions from the standard library can be used to manipulate data regardless where it comes from. Once you learn a few common patterns for transforming data, you can apply these patterns everywhere.

I strongly suspect that data driven APIs are a major reason why Clojure libraries tend to be so stable. When a library is simply transforming data then it's possible to get to a state where it's truly done. Once the API consisting of all the supported transformations has been defined and tested, then the API is complete. The only times the library has to be revisited is when use cases missed by tests are discovered or new features are added. This tends to happen early on during library lifecycle, and hence mature libraries need little attention from their maintainers.

Of course, this is not to say that large software cannot be written effectively using OO languages. Clearly plenty of great software has been produced using these techniques. However, the fact that complex applications can be written in a particular fashion is hardly interesting of itself. Given enough dedication and ingenuity it's possible to write complex software in any language. It's more useful to consider how different approaches impact development style in different languages.

October 3, 2017

Moving Forward By Letting Go

Lisps are famous for having powerful metaprogramming facilities derived from their homoiconic nature. Core language consists of a small set of built in primitives, and the rest is implemented using macros and functions in the standard library.

Since these same tools are available to the users, anybody can easily extend the language to add semantics for their problem domain. This is the aspect of macros that's discussed most often.

While this is certainly the case, Clojure community tends to discourage using macros if they can be avoided. The rationale behind this being that macros introduce additional complexity, and Clojure values simplicity. I generally agree with this sentiment, and I find that I tend to use macros sparingly myself. However, saying that macros shouldn’t be overused is not the same as saying they shouldn’t be used at all.

One place where macros work well is library APIs. Libraries that express a particular domain can create a DSL in their API that cleanly maps to that domain. Compojure, Specter, and Clara Rules are great examples of effective macro use. Such libraries are a powerful demonstration of users extending semantics of the language.

Since most ideas can be expressed via libraries, it becomes possible to experiment with different approaches to solving problems without modifying the core language. Extending the language through libraries has the advantage of keeping these extensions contained. It also lets them fade away when you no longer use them.

Clojure has been around for a decade now, and the core language hasn't changed much in that time. Some new features have been added, most notably reducers and transducers, but overall the language has stayed small and focused. In fact, I've even seen concerns that Clojure is stagnating because features aren't being added at the rate of other languages.

The idea of using libraries to add features is used by the Clojure core team as well. Consider the example of the core.async library that brings Go channel semantics and the CSP programming model to Clojure.

Perhaps, in time a better idea will come along, and core.async library will become deprecated. At that point developers would stop using the library and move on to use whatever happens to replace it.

Meanwhile, existing projects will not be affected as they can continue using the library. The community will move on, and most people won't have to learn about core.async semantics. This cycle may happen many times with many different ideas, without any baggage being accumulate by the language itself.

Unfortunately, mainstream languages are designed in a way where it's not practical to add new features without updating the language specification to accommodate them. Popular languages such as Java, Python, and JavaScript have accumulated a lot of complexity over the years.

As usage patterns change, new features are being added, while existing features become deprecated. Removing features is difficult since many projects in the wild end up relying on them, so they're typically left in place.

Having lots of features in a language can seem like a positive at first glance, but in practice, features often turn into a mental burden for the developer. Eventually languages become too large to understand fully, and developers settle on a subset of the features considered to be the current best practice.

In my opinion, this is the real power of homoiconicity. A language that can be extended in user space can evolve without accumulating baggage. New ideas can be implemented as libraries, and later discarded when better ideas come along. The end result is a small and focused language that doesn't sacrifice flexibility.