After several years spent building technology at the bottom of the stack (microcontrollers, assembly programs, distributed optimization algorithms), I have now been in the application space for a decade. I am kind of bored.

App Hype

Building an app is a very fun process. It’s fast-paced, intriguing and goal-driven. Three great qualities deeply rooted in today’s businesses.

The first app I built was a Django app. Django is a web framework, which means it provides tools off-the-shelf to develop a web application, and it is developed on top of the Python programming language. So, my first app was a web app, which I believe holds true for the vast majority of app builders.

It wasn’t for work, in fact it was a personal project. As of today, I still think that the app was great. It was a small website that people could use to analyze pieces of text, such as articles and essays. In effect, that was my first step towards becoming a writer. I called the website YSEO, a horrible name that to me meant Your SEO. I have two friends who work in the SEO space and my ideas about that app were influenced by my conversation with them. Yet, the name was awful.

Technically though, the app was really great. There was a mix of (basic) NLP, web API and UI that made me think it was a nice piece of work. Or maybe, more simply, it just was my first app.

The app worked well. I deployed it and made it accessible on the internet (back then, that was my first exposure to development operations) and, of course, nobody ever used it. Not even once. I took it down and never fired it up again, though I still have the code. The best part of it, in hindsight, was the lot of things I learned about the tools I used; they were very useful in several paid projects down the road.

Today, several years later, the things I remember most vividly about that first app are: the nonsense rush towards building the final shape; the many interesting technical details ignored because of that same rush; the nonsense technical choices due to the skipped technical details. And when I think about all that, I regret not having dedicated more time and not having taken a real engineering approach.

This is the sad reality about attempting to do engineering in a rush: it’s not engineering anymore. Writing an app is not engineering, it’s code writing (in fact, like we all know, it’s more copying & pasting than writing).

As experience grows, past mistakes can drive decisions while building new apps. In this case, it’s definitely possible to avoid many pitfalls and write some decent piece of software after all. This is a major reason why some developers write apps better than others: they have made a lot more mistakes in the past, because they have attempted more times. Like they say, practice, practice, practice.

Learning from mistakes is not easy by any means, but if you keep your brain awake and reflect on what is (and was) happening (a process that is called metacognition), then you will indeed learn.

Anyway, apps are written out quickly and engineering is sadly left behind. Every now and again I look at some apps that I wrote a long time ago and I always think “How could I make such bad decisions?”. And I am not talking about silly mistakes in the code (those happen all the time), but plain poor engineering decisions. Do not get me wrong: my apps usually work pretty well. But they rarely show that there’s an engineer working behind the scenes, because engineering is a different thing.

Engineering

Engineering work is novel and intrinsically requires human judgment. It produces a permanent improvement in a service, and is guided by a strategy. It is frequently creative and innovative, taking a design-driven approach to solving a problem—the more generalized, the better.

I found this beautiful definition of engineering in Google’s book about Site Reliability Engineering. Let’s talk about it.

Engineering work is novel, and I couldn’t agree more. But all apps look the same to me! Take the user, save the user. Read the item, update the item. Now, on the one hand I am purposefully exaggerating a bit, but, on the other, the ultimate goal of “batteries-included” frameworks (such as Django) is exactly to take away a lot of the novelty, and basically guide the developer through a series of well-known steps, always the same steps, that lead to the final result: the app.

In other words: you don’t need an engineer to build an app. You need an app developer.

What’s the difference between the two figures? Here’s what I think. The act of coding becomes repetitive more often than you’d imagine. I am talking about the patterns in the action of typing lines of code; it’s more and more often always the same “kind” of code.

I am afraid this is a concept that only developers can agree on, but it’s a fact. Believe it or not, writing computer code can become very repetitive, especially when it’s “high-level” code (that is, you don’t interact directly with the machine). Back when I was writing assembly code for Atmel microprocessors, there rarely were two pieces of similar code. And if there were, I’d better put them in a macro because each block of code is a lot of effort! But today, at high level implementations (a category apps are part of), the mind of a developer goes more or less like this:

Mm OK, I will get this object from the server connection? How, who cares; it comes in this shape. I have to change this field and then call update, so it will save it. How? Mm. Who cares.

Then, in the next app she will write:

Mm OK, I will get the user profile in this format from the web server? How? Yeah I don’t know, but I’ve done it in the past, so… fine. Let me find that old piece of code. Oh, right, I just need to call update on it. Great.

Now, I think patterns are good things to have, even necessary. Patterns are the foundation of engineering (all kinds of engineering) and prevent people from doing things at hazard. Thus, I think that when it comes the time to apply a well known pattern, that’s a good thing. But how does the pattern fit in your system? Did you stop for a while and think about the overall design? Did you stop and think about how this stuff will evolve in five years and if it provides good enough abstractions? Did you stop and look at the whole from a few different perspectives, trying to imagine different usages, maintenance costs, fault tolerance and reliability? These and many other things are what defines engineering and they need to foster thinking.

As I was talking with a front-end developer in my team, a very skilled developer, he mentioned to me he wanted to start doing back-end because “front-end feels like always the same stuff”. I reassured him that back-end can indeed be a tad more diverse, but not by a great margin.

After all, software development is a standardized field. You must respect some standards for the result of your work to be good. But then my question is, when do standards start killing creativity? If REST is a standard, what’s the novelty in one app? Where is the creation there? If ORM is meant to make you forget about SQL, what’s the invention there? Is forgetting technologies and languages a creativity boost? Like you’ve guessed, I am deeply convinced the answers to all such questions is no.

In several books (Drive and Make it stick, to mention two) I read that development and engineering is considered to be creative work. Is that entirely true? The authors’ point departs from my ideas here, in that they assign those two jobs to the “creative work” bucket so to convince the reader that engineers and developers shouldn’t be controlled at work, they should rather be engaged. For all intended purposes, they are right and I agree with everything.

But, technically speaking, is development still a creative job, today? Go back a few decades, and they were discovering new ideas every other day. That was creativity on steroids. Today though, I think we are at the opposite end of the spectrum. If you asked 100 developers, how many would tell you that they do creative work?

I do a lot of development these days, mostly writing code. I think I may average a couple of thousands lines of code per day (in scripting languages, of course). I usually don’t regard those as particularly creative. Actually, it’s often the same jazz. So my criticism here comes from first-hand experience. And although I am struggling here to explain the exact feeling of writing always the same “kind” of code, because it’s indeed a feeling, and not something tangible, I know as a fact that most developers would agree with me.

If you look at software today, through the lens of the history of engineering, it’s certainly engineering of a sort—but it’s the kind of engineering that people without the concept of the arch did. Most software today is very much like an Egyptian pyramid with millions of bricks piled on top of each other, with no structural integrity, but just done by brute force and thousands of slaves. —Alan Kay

Alan Kay is not the average software engineer for sure, and as such he may be looking a bit at everybody else like pushpapers (or, well, slaves like he said). While he, in the sentence reported above, complains about the lack of vision of developers that led huge projects to implode, the same sentence offers me a different perspective. What it’s saying to me is that creativity lies in engineering.

Creativity lies in engineering. The act of design, that’s where the human brain starts spinning fast, then faster, to find creative and novel solutions. That’s also the human judgment that is “intrinsically required” to do engineering. Thus, engineering is design. But also architecture is design. Making music is design. Writing is design. Psychology is design. Thinking about people and life, that is true design. In fact, thinking, and just thinking, is design.

Yet, thinking is often neglected in development. What that means is that people have the horrible habit to jump at whatever first solution seems to be good enough. I believe it feels good to some, because this approach allows them to leave the uncertain terrain of design and move into the solid-grounded field of implementation. The design times are fuzzy and cloudy. There are endless possibilities that must be evaluated not only as good or bad in itself, but as good or bad for whatever scale the project will require in the next decades. Design is a space and time of open-ended questions. Some don’t like it. But I am thinking: isn’t that the perfect terrain for creativity and engagement?

Some want to escape that terrain as quickly as possible and start doing. It’s funny to say, basically some people like thinking less than doing. It’s a legit preference, but maybe, just maybe, it’s not the place for engineering?

If you want to be an engineer, do yourself a favor: always ask “what’s another option?”. Then, when you get an answer, ask it again. And again, a few times.

Options evaluation is all we are left with. With the enormity of off-the-shelf tools out there, some of which are undoubtedly great, oftentimes the technical tasks are almost automatic. And repetitive. But thinking about options, questioning them and going the extra mile to generate more options, and then spending all the time needed to evaluate them and to try to “view” the system several years ahead in future, that’s a creative effort that cannot become repetitive because it’s the pure act of thinking.

How can it be that thinking is neglected in development? I am afraid you can understand this only if you have actually worked in software development. Basically, the developer brain enters a lethargic loop where it knows exactly what she has to do, because it’s well known patterns. And because it gives a sense of mastery, it feels good. And because it feels good, there’s no incentive to change. I call this the hamster-writer loop.

Ah, Make it stick! The one takeaway from that book is that real mastery does not feel good. Mastery is three things: mindset, pain, asymptote.

It is a mindset, because you are what you believe to be. It’s an asymptote, because you cannot really ever touch it; you can only get closer and closer. And it’s a pain because it’s effortful—and here’s the key point. Mastery that feels effortless isn’t mastery at all: it’s an illusion of mastery. Mastery means constantly working one or two steps beyond your current abilities and, as such, it feels painful. How are you going to improve otherwise?!

What happens way too often in development is the lethargic status I mentioned above. Brain knows what to do. Brain feels at ease. Brain starts doing, thus skipping additional research (effortful), additional learning (painful), options evaluation (confusing, effortful and painful!).

I know this sounds crazy—how can a developer NOT think? We can. It’s a mix of experience and knowledge and, as such, is a very good thing in itself. But it’s static: it keeps you at the same level you are right now.

Engineering must always be novel, challenging, effortful.

Put a reminder for yourself: before doing anything, stop ten minutes and ask what are the other options. That’s the one trick that stops the lethargy to happen and that I’ve found works for me.

S.O.S — Space missing

So, I badly want to go back to technology, design and engineering. But where are the opportunities?

I worked for years in large enterprises. It’s really hard to do it there. I suspect there’s a big loss in translation through the various layers of hierarchy. I, like most engineers, have often wondered who the heck put some powers in project managers hands. Tough job, for sure.

Now, very big enterprises must be doing engineering, aren’t they? Actually, I’d say that they basically define what engineering is, today. Think about Google inventing the field of SRE and writing the de-facto book about it. Think about AWS writing the well-architected framework pillars’ whitepapers. So, how is it to do engineering there? I haven’t had first-hand experience with the very big ones (the largest I’ve ever worked at was Motorola), but I suspect that’s where Alan Kay’s sentence applies the most. Hundreds of developers put thousands of bricks one on top of a few others, in a race towards building a castle that stays up thanks to a huge workforce, not thanks to structural integrity.

The fact that they define engineering makes it less creative than it should be. Because they define it in terms of person-hours. Now, tell me that is the measure of innovation. What about vision, then? They need to have a great vision, needn’t they? Not so fast. They have business vision: for sure they didn’t build what they had without vision. But engineering vision is different. It’s about technical vision, is about viewing with the mind’s eyes the countless ways the architecture will scale, and figuring out the best design. It’s nothing about foreseeing users’ usage, pricing, sales.

I worked for years in the university as well. Things are no better there, in general. I am convinced that if you stumble upon some of the very inspired research groups then you can do real engineering. I mean, maybe it even becomes tinkering. But if you take a random sample of research groups, like most of the ones I’ve met, the odds are that they do things in a rush exactly like enterprises.

There are many reasons for this, not last the famous issues with the publication scores. I’ve had enough of this problem in my life, so much that I tend to avoid speaking about it altogether. On one hand, it’s a real shame. On the other, I learned to regard scientific publication the same way I look at musical albums: to find a good one, you need to listen to thousands. In research, to find a good paper, you need to read thousands because most of them are published just to fill the resumes.

Anyway, unless you’re very lucky and talented and hence have the chance to work with one of the few truly great research groups, then forget about doing engineering in the university.

Then, there are startups. I worked with all kinds of startups. Very tiny ones that failed after 12 months. Visionary ones, built by great people with great ideas, so great that they go straight their way forgetting a bunch of stuff along. And “power startups”, if you can pass me that name, that means well built organizations with good people and good fundings.

Now, about the first kind, they have no idea what engineering is. They don’t build stuff, they just use things like pieces in a puzzle hoping to make a pretty one. Which is a perfectly reasonable business model, but exactly the type of venture that provides little intellectual motivation to engineers. Enough of coding the same web apps over and over!

The second kind is a difficult one to approach. I was lucky enough in my life to meet one such startup. In general, it’s a slightly chaotic environment where two basic needs fight each other: the need to survive, hence to generate revenue, and the need to be noticed by others, hence to do something good, original and interesting. Like many things in life, this fight produces mixed feelings. Sometimes it’s all very exciting and engaging. Other times, you realize you must tell a slightly different story than the real story, because you need to generate money and get attention. As a matter of fact, engineers generally don’t like that. They are rational, fact-driven personalities. If they built A, they don’t like you going out and say we have built A and B.

This second type of startup has serious possibilities to do actual engineering. It just won’t happen in every project, and when it does then it will be for a shorter period than you wish. The urgency to sell is too strong.

And then there’s the third type. These startups basically are real companies, just with a small number of employees. They even have process and formal procedures like large enterprises. In my experience, most of what’s true for the second type of startups holds true also for this type. But the somewhat minor urgency for money can play a good role, at least initially, in that a stronger accent should be put on the design efforts. If that happens, take your time to evaluate the options. Don’t rush just because it’s nice to do stuff and it feels busy. Stop and think!

Being a middle ground between an enterprise and a startup, the third type gives some opportunities to stop, think and do engineering, instead of just app writing (which is anyway a central part). As the venture grows, there will be even more chances to work on serious processes, with the flip side that sometimes you may feel overwhelmed by the process and wish you could return to write some standard code without worrying about the whole system design. Here’s what I think in these situations: it’s not a worry, is a pleasure.

So, on the whole, the startup world is too heterogeneous to draw any conclusions about the quality and quantity of engineering done. How many startups are “just” web apps? A whole lot. How many actually spend time in design and engineering? A whole lot too.

With all that being said, the question still stands: where are the opportunities to do engineering?

I often wonder how nonprofits fit in this picture. I have never worked with a nonprofit. My intuition says that nonprofits might offer a middle-ground between revenue urgency and product quality—after all, they should not be driven by money, but they need money to run.

Who decides where the balance is? The people there, of course. It is always about people, eventually. Nonprofits are supposed to have a mission, but nonprofits also pay salaries, sometimes large ones. A nonprofit that does engineering may have a very good cash flow, and be concerned more about that than their official, original mission. Who decides that? Again, it’s the people.

Precisely in these weeks I am attempting at starting a nonprofit whose mission is education. Everybody’s education, sure, though that’s not the main mission. Customers’ education, yes that too, but it’s not the main purpose either.

The mission is the employees’ education. The organization’s mission is to make sure that whoever collaborates with it becomes more knowledgeable, keeps learning every week and, overall, becomes a better person over time. Think about it: doesn’t it ensure that everything is well thought before being done? That everything is engineered and designed, instead of just…done?

People, eventually

Thus, at the end of the day and after a long detour, we are back to people, because people make the difference even in this case.

The sense of dissatisfaction with repetitive, not challenging work can be alleviated by interesting conversation with people. And that’s not all; I am convinced that when such conversations happen in advance, then there’s no tedious work at all.

When the right people are involved, they will not consider anything that can be less than engaging. Engagement, not compliance, is what drives novel and creative work, such as engineering and design. In other words: when you are spending time with the right people, then everything, work and tasks will be engaging and lead you to a state of flow, because naturally you and the other like-minded people will select things to do and discard things to not do.

People in the workplace matters. If your boss is a control freak, then consider leaving. Great innovation is driven by engagement, not by control. Intelligent, engaged people are self-accountable and do not need to be controlled. This model of work is now a reality, so what I am saying should sound neither new nor innovative. It’s just a fact that you need to be engaged, for doing great creative work.

What has that to do with my previous rant about engineering and apps? Well, an organization with engaged people will understand if you candidly say “I am bored of this stuff”. Your boss must understand it as well, and if she doesn’t then consider switching to a different team. I know that’s not always easy, but I am convinced it is worth it.

It’s definitely possible to do serious engineering today, despite the hype about apps, the trillion bricks of code poorly written and the general lack of understanding about engineering. What are the options?

Back to posts