I sat down the other day to do a simple thing. I was going to write a C# implementation of a distributed Map/Reduce computation engine, akin to the Google implementation that they use to do “hard” things.
Coder speak: you provide two function definitions that mimic the Lisp commands Map and Reduce. Applying Map to a large data set is an independent operation for each element, and Reduce can be used to combine the answers of each computation into intermediaries and then finally into one answer. Splitting up the dataset to multiple machines and sharing the Map/Reduce functions makes it easy to do large scale distributed tasks. Hide that in an computation engine and boom, money and fame.
Non-coder speak: Google wrote about a hard problem, and I’m bored and want to do it.
But as is generally the case in computer science, the Map/Reduce problem is a specific example of the more general problem of designing a distributed system that can execute arbitrary C# code. The Map/Reduce problem would just be a specific example of a program able to run on the general system. But really, when I start talking about C# I am just using that as a placeholder for Lisp, a functional programming language that is much more suited to the task. So I should implement it in that. But maybe I could make it so that the execution of a language is defined in a grammar, and just write a parser and interpreter.
Somewhere shortly before my head exploded, I stopped. What had started as a simple enough program (not entirely true) ending up being rewriting an operating system and compiler, all distributed. That’s not just reinventing the wheel, it’s reinventing the Apollo program. Sure, it might be a good time, but it’s a nasty place to end up if you were looking for a casual programming project.
The problem with programming is that in almost every case the program you need to write is a special case of a more general problem. Whether it be a specific business need or a specific scheme of communication, generality is always more attractive than special cases to programmers. It’s more attractive to philosophers as well.
A student cheats. Why, and what should be done?
He cheated because he was out late at a party and doesn’t know the material, and if he doesn’t do well he’ll flunk the course. On a fundamental level he knows that the results of his test scores benefit him in the long run and that the time spent studying distracts from his chances to secure a woman to be his …. how do you say … “baby’s mama”. If he can manage both with the minimum effort, he has maximized his chances at reproductive success by giving himself more time to procreate and a larger monetary means to support them so that they too can be successful. Even more fundamentally he is acting in his own perceived self interest because all humans and animals are inherently selfish creatures. This is the only way it can be because of the law of survival of the fittest. Darwin is God.
So we let him go, right?
Nope, he flunks, cause that’s the rule. But it’s hard to keep your eye on the ball if you just generalize away reality.
Leave a Reply