good morning coaxidleglobe. today i will be talking about things i don’t really understand.
capital misallocation? let’s talk capital misallocation. if we are biased towards a radix ten - wait, what do i mean by that? well you’ve heard the joke “every base system is base 10” right? one-zero? i have to use the word ten not the numerals. and radix sounds cooler than base. as for bias: lets say an increasing the exponent by one is a recognized by all members of a population as something significant. it should change the behavior of what the population does. “an increase of an order of magnitude” is commonly thrown around across SV types. the bias is in what base you choose for your exponential. if we meet intelligent aliens with four or six phalanges on each hand instead of five, and they base their counting systems on that, the events they deem significant and all re-orient themselves around will be at a different pace than ours. my aim here with these lines of thought, by the way, is to try to get the human to not be so human-centered in thought. i would like to treat these entities that haunt datacenters as, if nothing else, capable of thought (on par with humans? maybe?), so in our own human thought we should accommodate these entities, right? the worst that can come of this is some thought exercises, right?
anyways. all the investing in “software” ever since the dot-com bubble has been one of the worst capital misallocation events of all time, by an order of magnitude (in radix ten) greater than the current “AI”-spending craze. crazy, right? “but it’s generated so many returns” that’s not what i am arguing against here.
i like computers. i have fun programming the computer and i think it’s interesting. but let’s do a little exercise going by a heuristic here, let’s the say purpose of these fields are more or less the first thing that comes into mind. if you wanted to get a little more fancy with it, you can do some nlp to see what comes up with, but let’s just be reductionist for the sake of it for now. the purpose of the internet is communication. “what about video streaming and payments processing and” stop there, we are being reductionist on purpose here. the aim of cybernetics is to introduce automata, or machines that act alive. the purpose of the computer is to take a set of symbols in, apply some transformations, and output a different set of symbols. “but what about when i drag my mouse across my screen” relax, that is the same thing going on, your computer just added on a little thing on top called a loop that means it continuously takes symbols in symbols out. not quite I/O like a lever you’d pull, but it’s all I/O.
let’s keep going down this line of thought and just go with the two questions “what is the purpose of the internet” and “what is the purpose of computers”. “whatever you want it to be” is the wrong answer!!!
“but those are just pieces of trivia, and not indicative of more important things like job performance!” you might object. i’d argue it’s pretty important in general that people understand the fundamentals of whatever their job is. but lets explore what happens when software engineers haven’t internalized these pieces of trivia, and instead are measured by workplace value and job performance. we give out these high salaries to people so they can stack layers of bullshit on top of other layers of bullshit. they are so clever they are able to move pieces of things from here to there. i’m going to point out to you a few places where ignorance of these reductionist fundamentals causes extreme waste.
this one is just awful. hundreds of thousands of careers (measured in orders of magnitude with base radix ten, of course) have been done around this. escape the json and xml and sanitize your sql inputs and length prefix everything all nicely. put the href inline in your http and then do an O(N) scan over your HTML document to find all the hrefs again (that have to all be quote-mark separated, by the way) so you can build a graph of all the relationships between different html documents. And we are going to serialize this graph in a bunch of sidecar files or just coordinate pairs or, god forbid, MongoDB (this one on its own is worth $22B).
you want to execute some arbitrary code on someone else’s computer without being able to have control of it? great, here’s fucking javascript!
engineering is a lower mental activity than science. it is somewhere in between a craftsman and a scientist. but when doing engineering, engineering is more or less about understanding tradeoffs. when programming reprogrammable hardware, there are some very obvious tradeoffs that need to be done, and that is why software engineering is a thing and isn’t a nonsense phrase. the reason why software is a thing in the first place is having to make a completely different chip every time you want to do something different is extremely uneconomical, so having a computer parts that can perform different things reusably is preferable, and the cost shifts from having to re-create the computer parts to having a human re-program the chip. one of the fundamental tradeoffs of software engineering is the trifecta of disk, memory, and compute. want something to query something? if you want it to persist (especially if your computer might suddenly lose power), save it to disk, but if you need it quickly queryable (which adds up if you have to query many things), you might want to keep it in memory via a daemon. relatively recently in 2006 there was a tool that came out called redis that could do both. have a lot of things to compute? you have to move from disk to cpu to memory and your results back to disk. have a lot of things to compute? use parallelizable cpu cores. if you have a bunch of sequential simple operations that can be done quickly and predictably, you might want to keep those all in a l3 cache, and you can morph your data into a way that looks like this but you might lose some definition, or if you want to do it slower but not lose that definition you can offshore it to the heap (memory). your tradeoffs depend on what you are trying to do but you can see how this turns into an engineering task.
scripting generally is not engineering, and most “swes” are engaged in glorified scripting. 90% of most professional software engineers aren’t really doing software engineering. instead what they are doing is messing around with building blocks that have been pre-configured by minds greater than them, as is the case with javascript, python, and ruby. all of these scripting languages all had this as their intended purpose during their creation, which should tell you something, and the other half is that these were intended to be general-purpose. some of these hit the mark more than the others. stupid tradeoffs like “do i want this to work on a machine that’s not mine” vs “do i want this to be able to do anything that’s not moving one value around from here to there” as is the case with python and javascript is NOT engineering. you are literally making a decision of “should i accept this language designer’s intended handicaps or this other one’s”. and to clarify, i don’t think that scripting languages are bad per se, they are useful when you are doing something quick and dirty and you know the rest, but there is something extremely alarming that we have conflated doing economically valuable work with doing merely scripting. on the flip side, sometimes these scripting languages are not meant to be general purpose and are meant to swiss-army knife be the right tool for the job and then somehow escaped containment, as is the case with PHP, which is a language i actually adore. optimizing your sql join statement so that it’s not doing quadratic lookups is not the same thing, that is a pretty straightforward optimization on a one-dimensional axis (query time) and is not trying to find some clever hidden goldilocks place between three axises.
a lot of this essay is a critique of how our internet works in general, and the capital misallocation i am talking about is just how much we have invested in it. linux, apache, and the world wide web all came up hand in hand (with the ideological roots laid down by richard stallman and the GNU project), and it is a little ironic that it took a physicist to come out with something that everyone in the world would end up using, and it didn’t come out of the unix cohort that preceded it, but it is clear that the design of this things was made by someone who didn’t understand some of these things that it suffers from, and we have paid dearly for it. the original unix cohort probably didn’t have enough of a carrying power to get something to literally a general audience, but it is kind of impossible to imagine a “what-if” the unix guys had got this into the hands of the general public. the chief grievance of how this all works is that is that separation is done by ascii. and this mistake has lead to inestimable man-hours of labor that were far too high paying for what they were worth.
want a reusable gui? we had something, it was the windows xp toolkit or the fucking java toolkit or fucking emacs, but NO you have to go ahead and mix up your presentation, scripting, markup, and styling all into one single text file with hyperlinks that aren’t really hyperlinks over a degenerating name addressed network. and i’m not a 100% cli/tui purist, although i really do love it and i prefer to do many many things in the terminal. when i want to watch a show i have downloaded late at night and i’m tired, i would rather click file > open and then click play and pause rather than have to remember a bunch of commands and hotkeys for a program i’m not using literally all the time. guis aren’t the worst thing in the world! just like how object-oriented programming really only is necessary for 10% of coding tasks, a gui is acceptable to use in about 10% of programming tasks. linus torvalds uses a gui. really i would prefer to browse the internet with just my keyboard. in silicon valley in 2024 there was a bit of a meme for something called “streamlit” which is like how do you get something like that popular when it is yet another layer on top of the cake. these gui frameworks, that microsoft windows and sun microsystems spread around, all started in these little firms that all passed them around internally and then had so much passing around going for it that when they opened it up to the entire world the rest of the world ate it up almost entirely all at once, which generally only became possible because of the internet (i mean, not just for software).
basically, we got really lost along the way. we abandoned project xanadu for HTTP which was a huge mistake. we let this physicists’s side project go all the way and now we have to pay with our time and money for it.
there were some stupid experiments with this god-awful invention called XML which tried to be a self-descriptive format but was still doing it in ASCII, a binary encoding schema which is in fact the wrong radix to be attempting to do things like that, and there was a whole mess downstream with that like RSS. yeah, congratulations, RSS and a thousand other SCHEMAS - not even a domain-specific language, a SCHEMA - that made a whole different mess of things. it’s not like computer science was in a bit of a drought in the early 2000s, CANs were invented back then, but for some reason the mainstream internet was doing something stupid.
measuring the capital misallocation is a bit of a simple task. you have to go with the census’s department’s estimate of how much has been given to “software engineers” and you will see that it is in the trillions.
the anglo socialization instinct that americans seem to have inherited, whether actual anglo or otherwise, is just all about this stuff. “oh i have a thing here i can do” “hi i make some bullshit i sell some bullshit to pass you some other bullshit” “oh really? well you’re in luck we are in the market of buying bullshit that we don’t understand and we have a really high budget for it because we don’t understand what you’re making” “great i don’t understand what you’re making you don’t understand what you’re buying let’s overprice the fuck out of this”. i hope you can understand why i am really snarky about this.