During my time as a professional software engineer, I have had countless debates over the merits of one design pattern, library, or concept over the other. Some are even publicly documented!

At this point in my career I have spent a number of years educating folks. This includes students in elementary school, to students in college, to hundreds of professional software engineers at previous workplaces.

Across these landscapes there has been one constant, you can tell people whatever you want, but often they won’t “get it” until they believe it.

Do by learning

Much of learning (especially in early school) revolves around telling students about why something is the way it is. A topic is explained in a very specific way. Students then work to understand it to some degree and memorize the rest.

This works well for most subjects. Ones like the biological sciences, history, or literary courses but for other more practical¹ subjects, this ends up creating dissent.

For subjects like math or computer science, students will often think up an alternative solution for solving the problem, often times thinking they “outsmarted” the teacher figuring out this alternative solution. Then they use it either because they are proud they thought of an alternative “nobody” has thought of before, or because it is easier for a small subset of problems.

While thinking up alternative solutions is important towards the advancement of these subjects, folks who are new to a subject will generally think inferior solutions (e.g. is only easier some of the time) or one that only works with a subset of the problems.

This isn’t always the case, but they are new to the subject! There is very little likelihood they will make breakthrough research before folks who are already established in the field.

Everyone has to learn math. This experience is so common it has become a meme. I’m sure most folks who are reading this blog have felt the same at some point. I certainly have myself.

Not just math

As a professional software engineer I am most in tune to seeing this when it comes to, well… software engineering work.

Experienced engineers will get stuck in their ways and often not take time to understand new solutions. New engineers (and beginners to coding) on the other hand will think they have outsmarted what their teachers and seniors have told them by completely reinventing the wheel or using a method that is easier in the short term, but much harder in the long term.

Telling these engineers the best way to do something is X is not enough. You can give every reason under the sun why a certain method is better, but this is generally not convincing enough. The new engineer will be resentful their way is not considered more. The old engineer will say this is the way it’s always been done.

You can bring a horse to water, but you can’t make them drink

The new engineer

On the python subreddit, every few days now there’s a new post asking about the use of globals. Until it clicks with the folks asking, their questions are always the same: well what if I change use of globals ever so slightly? Would that make them ok?²

More experienced engineers on the python subreddit have given the same advice again and again. Use of globals is quite bad except in exceptional circumstances. You can find the same advice on every³ single⁴ one⁵ of these posts.

The advice is always the same. This has been common advice for a very long time. So why do people keep asking it? In some of their posts these new engineers will even reference previous posts with the same advice they are about to be given, so it isn’t just that they can’t search.

For an experienced programmer, it’s trivial to understand why this is the best practice, because they have written extensive amounts of code before and understand why it is bad practice to use heaps of globals. But for a newbie they have no idea. They can put something in a global variable and for their script it is simply more convenient without seeing the downsides.

In this case the new engineer needs the experience. They need to be hurt by a hot surface and to pull their hand away. Rather than answering the question 100 times, the programmer needs to answer the question themselves.

A comment on reddit or a mentor explaining why something is the way it is will not be remembered nearly as long as a practical example.

If you’re teaching a new programmer, have the newbie write up a program that’s a few hundred lines with global variables. Then have them write the same program without global variables.

Make them not look at both programs for a week or two, then ask them to make modifications to them and it will become very clear to them why global variables are not great.

The experienced engineer

A couple of jobs ago, I was one of the folks who lead development on the internal Kubernetes cluster where all of our infrastructure ran.⁶

When running the cluster, we also had an internal docker registry where all of our proprietary images were stored. The registry situation wasn’t the best and it would go down from time to time. Nobody really owned it, so it was just intermittent firefighting that kept it up.

As someone who worked with Kubernetes a lot, I was well aware of these problems, and helped devise a system called docker-library for keeping images version controlled. So if the registry ever went down, we could rebuild them in a matter of minutes.⁷

While I was a huge advocate for docker-library and tried my best to get folks to use it, many refused to do so and preferred to build and push their images manually.⁸ Many of these folks very senior themselves. Some with 10-20 years of experience.

I tried my best to get folks to use docker-library, but few did. And then one day the worst happened, the single server where our docker registry was hosted died and we lost almost all of the images. Internal development ceased for a few hours while we did our best to bring the images back and restart failed jobs on the cluster.

We rebuilt and re-pushed all the docker-library images in about 10 minutes. We took hours trying to scavenge through build caches on local and remote machines to try and get the other images back. I ended up writing a Python script to search through Kubernetes hosts for cached images that were missing and then push them back to the new internal registry. The script was slow, but it ended up working and getting back 80% or so of the remaining images.

After this incident, the vast majority of developers started using docker-library to push their images.

Unfortunately they needed this both painful and practical learning experience to understand why this was important. No matter how much I advocated, real and painful practical experience was required to hammer this home.

Ideally if the company had cared about infrastructure, we would have had a red team to do this without grinding the company to a halt.

Learn by doing, don’t do by learning

Even when folks are trying their best to learn something new, learning by practical experience trumps all. Practical experience opens the door to learning more. Without that open door, nothing will get through to the folks you are trying to teach.

Practical experience doesn’t tell the whole story, but it creates the initial narrative about why material is taught a certain way or why a certain concept is important.

If you are trying to teach folks with an inkling of knowledge about a subject that can be explained with a practical example, make sure it is both included and that shows possible error states along the way.

Ensure you are teaching folks about why you are teaching something a certain way. Let them learn for themselves, and the rest will follow.