This post is not one of those blog entries where we list 20 examples from our codebase that make heavy use of algorithms and data structures. Well, we have those examples as well somewhere in the blog post, but that is not the point we are trying to make here. Read on, you will see what I mean.
D̶i̶a̶m̶o̶n̶d̶s̶ Fundamentals Are Forever
What do ASP .NET, Entity Framework, the Android/iOS SDKs (Software Development Kit) and node.js all have in common? They were all non-existent 20 years ago, and they will all likely not exist in 20 years' time—or at the very least they will become niche and obsolete technologies. That is because, as the world grows and evolves, so do its technological requirements. All those technologies that I just mentioned were not around two decades ago because there was no actual need for them to exist. And they will not be around in another two decades because there will not be a need for them to exist anymore by that time—the world will have different and quite possibly more complex needs come 2042.
- This is not news: decades might as well be centuries in the world of software engineering, such is the pace of our industry. And yet, despite this tremendous rate of evolution, there are still some aspects of our trade that are perennial—unchanged, and just as relevant today as they were when they were first invented, with their relevance likely to continue in the foreseeable future.
I am talking, of course, about the very fundamentals of computer science: algorithms and data structures (A&DS for short). They are the building blocks that keep our modern world ticking along and are responsible for all the modern creature comforts that we take for granted. They may not be glamorous and in the spotlight, but without them our world would be a whole lot different. Examples, you say? Advanced compression algorithms that allow you to stream your favorite shows. The routing and logistics algorithms that allow for same-day deliveries of your groceries. Your car's self-parking capabilities. Secure online banking and online shopping. The power-saving routines that allow your appliances to consume less power. The fancy code that does protein folding and chemical simulations in the hopes of creating better medicine. Your microwave, deciding how much and for how long to heat your pizza.
Everything in the modern world relies on algorithms and data structures, and this reliance increases with time—as technology becomes ever more complex and ever more ingrained in our society. What is interesting, however, is that no matter what "latest-and-greatest" technology you can think of you will always find that it is underpinned by the same principles and concepts invented long ago. And, make no mistake, the next generation of "latest-and-greatest" technology will inherit the same fundamentals. It is thus almost a duty for those working in the software industry to have a good understanding of these fundamentals.
Computer Programmer vs. Software Engineer
Now, make no mistake, you can get by as a computer programmer without knowing much in the way of A&DS. Given the modern frameworks and SDKs at our disposal and given the fact that a lot of software development these days is just moving data from one system to another according to some business rules, you can make a comfortable living as a software programmer without knowing much beyond what a "list" is and what the result of sorting it should look like. You can be the modern bricklayer.
Being a software engineer, however, is a different matter altogether. Engineering is not just building a house brick by brick; engineering is understanding why and how a house should be built in a certain way. What are the various trade-offs and implications of all your design decisions? What forces are acting upon each and every brick? What is the function of each brick? How will your house hold up in a couple of years' time? What can and cannot be done to extend its function and lifespan? You cannot answer these questions without understanding the fundamental forces and processes acting upon your creation. Sure, you can build a house quite easily with modern materials, but you will not really understand what keeps it upright. And regarding scaling up your construction, you might end up not with a skyscraper, but with the leaning tower of Pisa: both are relatively tall buildings, but one is a fraction of the size and complexity of the other, and I for one know at the top of which one I would feel safer.
The difference between the two buildings is not so much skill as it is knowledge. To build tall and safe, you need not only artisans to lay the bricks expertly, but also experienced engineers to determine where and how those bricks should be laid. Artisans alone will only get you so f̶a̶r̶ tall.
Who is a Good B̶o̶y̶ Software Engineer?
At the same time, all of this does not mean that you must inspect each and every brick and calculate each and every force acting upon your work. It just means that, as a good engineer, you have the capabilities and know—how to think about your work formally, when the need arises. This, you will notice, applies to all areas of engineering, not just software engineering. But limiting ourselves to the software engineering world, what does thinking formally about your work even mean? What does it involve?
What Makes Someone a Good Software Engineer?
There is, of course, the architectural component. The design patterns, the best practices, the coding conventions, etc. Most developers are familiar with these. Knowing these makes you an OK software engineer, but not necessarily a good one. There is also a deeper layer hidden behind these well-known and familiar concepts. It is the layer that makes the good software engineers, well, good.
There is a long discussion to be had here, but here is the gist: you need to think of your code both in terms of "classes, objects and API calls" and regarding "graphs, lists and formal operations". Each class is also a node in a graph of classes, each API call can also be formalized and thought of as a mathematical function, and the ebb and flow of what your application does is akin to the well-defined steps of an algorithm. In fact, your real-world application is just a practical application of theoretical algorithms and data structures. This means that, as a good software engineer, you can (and should!) apply the same logic and reasoning to practical software development that you apply to theoretical A&DS questions.
There is a beautiful and subtle duality here that, once you finally see, you cannot unsee. And being able to immerse yourself in this duality will turn you into a better software engineer. Does this involve memorizing by heart all the algorithms that you come across? Being able to write a working quicksort implementation on a whiteboard? Solving Olympiad-level problems while at the pub with your friends? No, no and maybe (kidding about the maybe). It is a lot simpler than that. To become a good software engineer, you just need to:
This does not mean that you must be a genius or a hermit living in a cave surrounded by ancient texts, performing arcane rituals. The barrier to entry is really not high at all. You will see later that there are really a couple of fundamental concepts that you need to understand—the catch is that you really need to understand them. And truly, blood sacrifice is optional.
So why go through all the effort, you ask? Good question, and this next section will try to give you some of the reasons why.
F̶r̶i̶e̶n̶d̶s̶ Algorithms and Data Structures with Benefits
It is a common pitfall to think that A&DS are theoretical only and have no impact on real-world software engineering. Nothing could be further from the truth! To exemplify, here are just a few unexpected real-world benefits of thinking in terms of formal algorithms and data structures. These apply regardless of your tech stack and position on the engineering food chain:
Much better architecture for your projects. Classes and objects are just nodes in a graph and applying graph-related thinking to your class hierarchy can yield surprising results. The same goes for your actual code; you would be surprised how much you can simplify your code when you apply formal thinking to it—and how natural it becomes after a certain point.
Way better code reviews. Scrutinizing code with a formal eye tends to bring out hidden flaws or weak spots, that you would otherwise tend to gloss over. Just because it "looks right" does not mean it is right and applying some mathematical rigor every now and then does wonders for your codebase.
Fewer bugs, more robust code. Not only do you have better architecture and better code reviews when you apply the A&DS mindset to the real world, the code you write is more robust to begin with. Because you reason about it and try to find counterexamples and flaws, and because there is a spider sense in your brain that says, "something is not right here, keep poking". And should you decide to cut corners (let us be honest, we all have to at some point), you will have a much better understanding of the implications.
A much easier time prototyping. This may sound paradoxical but applying the "A&DS formal mindset" to your prototypes is incredibly helpful. Not only do you tend to spot dead-ends early in the game, but you can also analyze your "what if" scenarios before you even write a single line of code.
Easier, quicker, and more productive conversations with your peers. When you all speak the same fundamental language and can view code abstractly, discussions and problem solving become a lot faster and a lot more fun. You can debate your code for one hour, or you could just say something like "this is similar to this-and-that concept/problem/algorithm". Not only did you just save everyone a lot of time, but you have also gained the benefit of all the formal work and reasoning already done on whatever "this-and-that" happens to be in your case.
Now, all of these are real examples, but it does not mean that you will begin to talk to your colleagues in mathematical notation or use Boolean algebra to analyze your code. Rather, all these positive behaviors and their outcomes will manifest themselves subtly and gradually. You will most likely not even be aware of them, but I can guarantee they will have a noticeable impact on your work. Furthermore, just like brushing your teeth, they will become second nature to you—even though you will sometimes forget or consciously choose not to do them.
101 D̶a̶l̶m̶a̶t̶i̶a̶n̶s̶ Misconceptions Around Algorithms and Data Structures
Let us look at some common misconceptions around A&DS—those that usually tend to scare people off at the mere mention of the subject. And relax, there are not 101 of them:
It takes a lot of time and effort to achieve results. I will tackle this one with a metaphor. Ever see a baby learn to walk? At first, his frustration is high, and his progress is rather slow. But he keeps at it, and soon discovers that being able to r̶u̶n̶ w̶a̶l̶k̶ stumble from point A to point B on his own is somehow fun and useful. Fast forward a bit, and he stumbles less and less, until suddenly he is walking like there is no tomorrow. Eventually he will be capable of a mild jog or even a short run, which is a skill that comes in handy (especially if you are a kid). And these are all skills that will stay with the tiny human for the rest of his life. Of course, to go from an immobile infant to the next Usain Bolt takes a huge amount of time and effort, and it is not for everyone. But to become someone that can walk comfortably and occasionally run—that is surprisingly natural once you get over the few initial bumps and scrapes. Just keep at it.
Algorithms and Data Structures at UiPath
Our Interview Process