Meta-Problems

The other day, I overheard one co-worker showing Emacs to another co-worker. The one who was new to Emacs said something like, "Oh, it doesn't do X?" The Emacs proponent replied, "Well, no, but you could always implement it if you want it."

And this statement sounded really familiar. It sounded just like Paul Graham's description of the Blub language (which is an amazing essay, I must say).

Now, I'm not saying some other editor is more powerful than Emacs. But I am saying that the fact that this statement was made is an indicator that Emacs isn't the most powerful. It's a Blub editor compared to the imagined ideal editor.

The entire idea of a Blub language and the power continuum doesn't apply to just languages; it applies to all tools which are powerful enough to emulate tools with more power. With programming languages, as long as a language is Turing-complete, it can emulate any other language that is Turing-complete. A similar "whatever you can do with your tool, I can do with Blub" argument can be used with editors when Blub can be extended with plugins or scripts.

Emacs may be powerful due to its extensibility, but I think we can do better.

Patterns of Patterns

Building on top of the most powerful tools available, which eliminates usage patterns and implementation details, leads to new usage patterns. Eventually, these patterns will be considered implementation details. When the Internet was first invented, HTTP GET wasn't an implementation detail; it was what they were creating — the churning fringe of development. Now it's a detail, as we're creating AJAX web apps built on top of HTTP and a slew of 57 other things that didn't exist before.

Whatever you make, there will always be patterns — patterns of repetitive tasks that are tedious, expensive, and could be done cheaper by a machine. Thus arises the desire to automate the task completely, or at the very least, create a tool to make the repetitive parts disappear. When this is done, the pattern becomes embodied in the tool, and at first, all patterns seem to have been eliminated. But in time, new patterns emerge, in which your tool is just a small piece. At this point, you're back where you started, possibly with more resources and personal growth if you're lucky. But the loop begins again.

Seeing this meta-pattern, we can attempt to follow it to its conclusion. If we're going to create tool X in the first tool-making iteration and tool Y in the second tool-making iteration, can we go directly to Y and skip X altogether? If you can see patterns 2 or more iterations ahead, you may still have to implement X but you'll have a great advantage by seeing the longer-term goal of Y. That said, it's hard to see patterns, even in this iteration, let alone iterations far in the future. Who knew, that in just 10 years, the web would evolve the way it did. Back in the 90's, it wasn't even clear that anyone could make money off of a search engine (one of the most lucrative fields) because the thinking was that people would search for something once, bookmark it, and never return to the search engine — a legitimate concern back in the day. Obviously wrong though knowing what we know now. ...So can we do better?

If you noticed, what I just did was unroll the tool-making loop 2 times. We tried to exploit the pattern by jumping ahead 2 loop iterations at once. But to fully exploit a pattern, you must look at the big picture. You have to look at the entire loop all at once.

Most people are stuck in 1 — the current iteration. Last paragraph, we jumped to 2. What about n? The "limit", if you will.

The first thing that sticks in my mind is the loop condition. When do we ever exit? Why are we here in the first place? There are many directions this line of questioning can take. One of them — if you pursue it faithfully to its conclusion by looking for the limit of the {meta to the power n}-problem — leads to questions like "What do I want out of life?", "What is my purpose?", and "What is my true essence that defines me?". But those are beyond the scope of this essay.

Another direction we can take is to observe the fact that as long as we are in this loop, we are struggling to find a pattern, factor out that pattern, and make the thing we originally set out to make (at the non-meta level). And this is all irrespective to what we're actually making. In other words, as long as you're making new things and competing for resources, regardless of the field, whether it's software, clothing, or underwater basket weaving, you will always be in this loop.

To What End?

When I was younger, I thought solving certain problems on the computer was valuable. I began programming to solve some of them.

But after a while, I observed the act of creating software as being repetitive. The act of programming — the very act of problem-solving — became the barrier to solving problems. The problem-solving embodied in programming became the new problem. It was a meta-problem because I didn't actually care about writing the programs themselves; I cared about the solutions that the programs entailed. Writing program after program, it only made sense to solve the meta-problem. And that is how I got interested in programming languages, the medium through which all programs were communicated.

Looking back, what I did was jump from 1 to 2. But for a while, I failed to make the quantum leap from 2 to n.

The human race will no doubt continue to improve technology ad infinitum, becoming more efficient and allowing people to do things they could never do before. And for short-term goals, this makes sense. But how is that any better overall?

I used to think that advancing technology to do work for us was a good thing. It would free us up to focus on what really mattered.1 But why not just focus on what really matters right now?

Some people will say that they can't because of some thing X that prevents them. To that, I would argue that you will have to learn how to deal with X, because even if you solved X (which humans will one day), there will always be Y to fill the role of the thorn in your side. Of course, some will say that advancing the state of the human race in the way I describe is what matters. And to that, I would say you've never honestly asked yourself "why".

When you're being practical in the short-term, the thing you're actually making on the non-meta level is what matters. But when you're setting long-term goals, the solution to the meta-problem is what matters, because the problem (just like in software) inevitably arises again in an infinite number of variations. It only makes sense to come up with a general solution.

But you can apply the same reasoning to the meta-problem itself, and think of it as simply another problem. Now you care about the solution to the meta-meta-problem, and it makes no sense to settle for a solution to merely the meta-problem. ...Ad infinitum.

What does all this "meta" talk mean? It means that technology is inane — it can't solve your Ultimate Problem. There will always be another problem. So should we give up and stop advancing technology?

It only makes sense to do something that has no inherent value if it works towards value elsewhere. So, if for example, you were using the day-to-day work of building the next great web app that allowed people to eat ice cream in a way that was socially networked (with a news feed of who ate what flavor combinations when (and the number of pounds they gained as a result (updated in real-time (and notified via text message)))), then go right ahead. Ditto for creating AI or reciting 1024 digits of π while juggling if you believe it is a means to something inherently valuable.

But to think for one moment that it in itself will satisfy you in the long-run, that it is a valuable end in itself, you're deluding yourself.

---

1. It seems silly to think we will one day have robot slaves doing all the work for us, but if you follow the current pattern to its conclusion, that's what will happen, among other things.

Stand on the Shoulders of Giants

I feel like I haven't even written a useful piece of code in ages because the mere thought of boilerplate code stops me in my tracks. This is one of the main motivations behind creating a new language free of hindering boilerplate code. But then you start running into other problems.

The egotistic developer will believe that creating a new programming language is the key, secretly striving for the silver bullet, even though he would never openly admit it because he is not even conscious that he is doing it. Some people believe that the silver bullet is Lisp, but for some strange reason, it has failed to slay the dragon for the past 50 years and counting.

The more seasoned developer, unambitious and static, will believe that creating a new language is a waste of time due to how many wheels have to be re-invented in a new language before it even approaches the usefulness of existing languages. Not to mention the fact that any feature could simply have been implemented in (or on top of) the existing language in the first place.

I think the only thing that makes sense is to strike a balance with something that allows (an order of magnitude) more succinctness and extensibility without sacrificing the millions upon millions of man-hours already spent by armies of programmers. We can stand on the shoulders of giants like IBM, Microsoft, Sun, and Google, and move forward without taking a giant leap back.1

On that note, we have people building things like Instapaper. And Instapaper is great; I use it. But it's silly that I now use 2 completely separate bookmarking services that are oblivious to each other.

Of course, upon closer inspection and further use, I realized that the way I use del.icio.us and the way I use Instapaper are completely orthogonal. That is, the set of links that I save to del.icio.us and the set of links I save to Instapaper are completely disjoint. Occasionally I will first save a link to Instapaper, read the article, and then save it to del.icio.us, but that is a rare exception.

However, it's completely obvious to me that the functionality of Instapaper is a proper subset of the functionality of del.icio.us. In other words, everything that Instapaper can do, del.icio.us can do and more.

Why then didn't Marco build Instapaper over del.icio.us? It seems like a perfect match.

With the Web 2.0 craze, APIs started popping up everywhere. Even tools like Pipes and Popfly to wire those services together. But who is using them?

Instapaper's value is solely in its amazingly simple interface. The reason I use it differently from del.icio.us is because its interface affords to different things. It makes different operations cheap, and that changes the way I think about those operations. But why doesn't Instapaper integrate with my del.icio.us account, simply tagging things with a "read-later" tag? When it comes to bookmarking, del.icio.us is king; there's no disputing that. Instapaper wouldn't have less value if it were built on top of del.icio.us, it would actually have more.

And this, in my humble opinion, is the next step for the web. People will finally start realizing that most of the functionality in their little web app to-be is already done — not in a library — but in a web service.2

Services that do single things — and do them right — will be indispensable to the web. And a glue language will be in high demand. But personally, as an entrepreneur, I'm not looking towards Yahoo Pipes or Microsoft Popfly (yes, in direct opposition to the idea of standing on the shoulders of giants). I want to own my mashups, and those services don't give me that at all.3

---

1. And this is why I have been very interested in Clojure — an extremely practical Lisp built on top of the JVM.

2. For example, I will almost never have to create my own charts, as Google already did it. Ditto for maps.

3. If you haven't heard me practically shouting this already, I want a glue language for the web! I will personally thank anyone who builds one that doesn't suck. But people, please don't comment here saying X is already that glue language. [Ruby on Rails people, I'm looking in your direction. ;-) ]

The Cost of a New Language

As I described in my post on tool-making, the most sensible thing to do is to minimize total cost to reach your goal. Accounting for all costs, it occurred to me that creating new tools such as programming languages can inadvertently increase total cost by reducing the usefulness of other tools designed to be used with other languages.

For example, gdb is an extremely useful tool for working with C code. But if you start using Java, the usefulness — or the amount of cost savings — of gdb is reduced so much that it is practically useless. What you really need is a debugger designed for Java. If you didn't have a debugger for Java, your total cost may have increased as a result of switching to Java. (I'm exaggerating of course, because it's not just gdb but other tools as well. And they all add up.)

I recently became aware of Google Web Toolkit (GWT) which basically compiles Java to JavaScript for use with AJAX web apps. At first this sounds right. Gain static typing and a unified server- and client-side language. However, this didn't sit right in my mind and I wasn't sure why. Then I realized it troubled me because, in a way, GWT was compiling from a lower-level language to a higher-level language.1 And the thought of that is just absurd. Once you take into account the reduction of usefulness of tools though that results from switching from Java to JavaScript, it all makes sense.

So when does it ever make sense for a small software company to create a new programming language? I'm beginning to think the answer is: never. Although there are inspiring words saying how much more productive a good programming language can be, there are also stories of people working with such languages their entire lives and how they became disillusioned.

The only sense I can make of all this is that people love to hear stories like Paul Graham's that hold the carrot of hope out in front; the idea of creating the ultimate programming language is extremely ego-boosting. But at the end of the day, if your goal is to create great software that you can live off of, then there are very real costs to using less popular languages, and the trade-off does not often make sense. GWT on the other hand, works because it enables rather than disables the use of well-developed tools of another language. A new language is just a tool, and it should help, not hinder, overall.

...Another burst of insight came to me. Say you created a new source language S but wanted to exploit the abundant, well-developed tools in a target language T. What if, when you wrote your compiler from S to T, you also wrote a translator for the tools. In other words, think of the API or data-structure that the tool works on as T', and create another compiler from S to T'. It all goes back to the idea that everything is a compiler.

I don't know if this is something feasible. Again, it's more of a what-if question. Is there a way to compile away the costs of a new language?

---

1. Another reason GWT seems weird to me is because instead of its source language being under-developed, its target language is.

In the future...

In the future, no one will be using test-driven development. Instead, we will be using proof-driven development. A programmer will create a specification for a program in a language like Gallina and prove its safety and correctness using a proof assistant like Coq. Once done, a compiler will transform the specification and proof into an executable program which is proven to be semantically equivalent to the specification. In addition, the proof will be bundled with the machine code of the program so that a third-party can statically check that the program adheres to its safety policy before running a single line of code, a technique called proof-carrying code.

In the future, machine code will not specify instructions to be executed. Instead, machine code will specify the layout of a circuit, and the computer when executing that program   will "re-wire" itself to match the specification. Data will flow through the constructed circuit as fast as physically possible. And when execution is finished, the circuit will be re-used for another program after another re-wiring. Of course, the computer won't literally move wires around; it will probably be more akin to gates making and breaking connections. How is that different from today's computers that use transistors as gates? These gates won't be clocked. Moreover, these computers won't be limited to running N things in parallel, one for each of N cores. The computers of the future will be more like today's FPGAs in that you will be able to run as many things in parallel as you can fit in your circuit, which could be on the order of thousands or more.

In the future, people will create life in their own image. They will start off with simple robotic toys for entertainment. Then they will begin automating mundane tasks like cleaning and maintenance. Then they will progress to having robots take part in high-risk situations like flying planes, performing surgery, and going to war. But eventually the technology will become so well developed and cheap that creating automata will be as easy as hacking scripts on a home computer. These little things will infiltrate our lives the same way PCs did in the 80s and 90s. And people will accept them. It won't stop there however, because people always want more; it's in our nature. And driven by money, we will create automata (they won't be called "robots", but something more marketable) to satiate our addictions. The female sex slave will come out first, and soon after followed by the male model. But with all these automata running around, people will begin to feel disconnected, and they will want to have someone in the lonely world to listen to them and comfort them. They will again turn to their ever-constant automata for support, as they are the only things that can be consistently controlled. So they will make automata that can please one not only sexually, but emotionally and psychologically. Of course, to do this, they would have to make an automaton that can empathize, an automaton that can understand the feelings that its master feels. And the only way to do this is to create an automaton that can actually feel these things, the same things that its human master feels, and be capable of going through the same experiences. But once this great feat is accomplished, people will then begin to realize that what they've created — with all their own feelings, desires, and experiences — won't want to serve anyone but themselves. Humans, in their infinite ignorance, will have created beings not only in their own image, but in their own problematic situation, in an attempt to solve that very problem. And in the end, we won't be any better off except in realizing that there is nothing fundamentally different between a human body and an automaton body.

In the future, every choice we will have made will have become obvious. And in the future, we will still believe that we are in control of the choices we make. In the future we will look back on the past with a nostalgic eye. And in the future, we will still believe that there are better days far ahead. In the future, we will continue to run and run towards comfort and safety. And in the future, we will not find what we are looking for until we give up looking.

In the future, I will see some more patterns. And in the future, I will sit back and do nothing about them. ...Unless of course — something changes.

The Greatest Contribution to Programming Languages

Thinking about tools, it occurred to me that the greatest contribution to programming languages is not any single language. No, not Lisp, C, ML, Haskell, Qi, or any of the more obscure yet equally great languages.

The greatest contribution someone could make to the programming language and compiler community   is not to make a great programming language, but to make the tools necessary for making new programming languages cheap — so cheap, that programming languages and the tools that allow their efficient use (e.g. compilers, debuggers, code-editors, etc.) become abundant.

And once they become abundant, people will start to see how interfaces are everything, how everything is a language for a compiler, and how there's no real answer to code's worst enemy — size — other than abstracting away parts of it and then specifying which parts you need in a particular situation via a language.1

What tools would make implementing new programming languages cheap? Let's make a list.2

• Parsing and Printing (Bi-directional Mappings between Text and Abstract Syntax)
• Tree/Graph Editor
• Inference Engine (Theorem Prover)
• Data-flow Analysis
• Incremental Transformation Engine (Rule-based Transformation, Incremental Compilation, Execution Stepper)
• Dependency Analysis
• Unified Runtime System (Memory Allocation, Garbage Collection, Runtime Safety Verification)

Some more details...

Parsing and Printing

This may be a first-step before a true code-editor is created, but I suspect that it will still be desirable in some cases to stringize code. In addition, I imagine parsing code, perhaps not entire programs but small code-blocks, will still be useful in a code editor.

What is completely obvious to a compiler-writer (in fact, I knew of someone in college whose side project was exactly this) yet others seem completely oblivious to, is that you can make "skins" for your code to format it differently in text, but have it parse down to the same underlying representation (i.e. abstract syntax). This way, Joe can use curly braces to specify blocks while I can use whitespace indentation, and it will represent the same code. The same for every other parsing variation, small or large.

What's cool about this is that even though Joe and I are typing in code differently, I can view Joe's code in my format and vice versa. So no more conforming to (practically uncheckable) coding standards enforced by your tyrannical employer.

I imagine specifying these via bi-directional mappings between strings and abstract syntax data-structures. In the past, I've always had to write one piece of code to parse and another piece of code to print, and that's just silly. Of course, there are corner-cases which are specific to one as opposed to the other, and I imagine some kind of declarative DSL for specifying these.

Tree/Graph Editor

Code at its core is a tree. Once it is parsed, it is represented internally as an abstract-syntax tree, or absyn tree. If you consider the semantics of the language though, variable references for example, refer back to nodes closer to the root of the tree. So in this sense, it is a graph.

The core of most IDEs today is a text-editor. This is silly though since the programmer is actually editing code, which is this tree or graph data-structure. In particular, when you edit the name of a variable reference, the editor should distinguish between changing what that node refers to and re-naming the node, as overwriting a variable in text changes what the node refers to; to re-name, you have to manually replace all references to it, including taking into account scoping rules of the language. The editor should understand when you're trying to wrap an expression in another expression, instead of relying on you to go to the beginning, add an open-paren, go to the end, add a close-paren.

Sure, it's a matter of convenience, but it's also a matter of error-prone-ness. I can't count the number of times I've had bugs due to copying and pasting textual code. If the editor I was using was aware that the structure I was copying was a graph, it would indicate to me which nodes the pasted code were referring to, allowing me to see my bug immediately.

The editor must be aware of the semantics of the language you are editing and give continuous feedback of changes made. If it weren't aware of the semantics of the language, how would it give you feedback on what you are editing that is specific to the language you are using. Thus, your code editor must be tied very tightly with your compiler. However, a general tree-editor would be a huge step in the right direction.

Inference Engine

An inference engine is critical for any code-level automation. For example, type checking, type inference, and semantic-preserving optimization. This is no simple feat to create, as it is, at its core, a theorem prover.

Instead of re-writing a theorem prover for every compiler, a compiler writer should only need to specify the axioms and rules of inference for his language to the industrial-strength theorem-prover. Simple things like constant folding and more complicated things like induction-variable elimination can fall out of this.

Data-flow Analysis

Of course, some standard optimizations like dead-code elimination (if your language is eager) and semantic checks like whether variables have been initialized (if your language allows such things) rely on data-flow analysis. Some compilers even use this to optimize memory allocation, as in whether to allocate something on the stack or the heap, by determining whether a reference to an object may escape a local block.

Again, this is another instance of something that can be done once generally, but it should also be tied in with the incremental transformation engine to allow for only re-compiling or re-executing pieces of code that have changed. This could also be extremely useful in a static sense where the editor displays to the programmer what pieces of code are affected by the flow of changes made (since the last save or last revision for example).

Incremental Transformation Engine

This is the meat of the compiler: the transformation engine. It transforms source code to object code. Or in the case of an interpreter, source code to immediately-executed instructions. It must be incremental in the global sense in that it must handle incremental building for a more responsive development environment. But it must also be incremental in the local sense in that it must allow for the programmer to pause, step, observe, or otherwise control the transformation for debugging purposes.

Home-grown interpreters usually start out all-or-nothing; they either transform or they don't. The reason is simply because the most straight-forward and clean implementation is a recursive traversal of an absyn tree. This leaves no room for stepping through code and displaying the current state to the programmer. A debugger is an after-thought, and rightly so, as its cost outweighs its benefit when a language implementor is creating a language for himself or others who have access to the language the compiler is written in.

I imagine writing pattern-matching rules for the transformation engine similar to the way someone might write a recursive traversal of an absyn tree in ML, however, the transformation engine will automatically handle incremental stepping and observing of the transformation process for interaction with the programmer when debugging. In addition, I imagine many additions to this language compared to ML which allow short-hand for specifying common cases of transformation, like only specifying the differences in base cases and the engine recursively applying the transformations to all other nodes.

Dependency Analysis

Based on dependency analysis, the transformation engine can automatically determine which code to (re-)compile and in which order. This should include external dependencies as well.

In many projects I've worked on in the past, circular dependencies have prevented standard tools from working as expected. As a work-around, we were forced to transform our module layouts and handle the dependencies manually. Although circular dependencies in general can lead to confusing module-interfaces, there are some cases when it just makes sense, and the transformation engine should handle circular dependencies by simply using a two-pass algorithm.

Unified Runtime System

This has got to be the most depressing topic for all language implementors. If you implement your own runtime system for memory allocation and garbage collection, what hope do you ever have of your language interacting with the rest of the world — something that is necessary both for production apps and popular adoption. If you use an existing runtime system, you are stuck on its platform and forced to use its (often unsuitable) virtual machine.

Parrot, looking good. Perhaps this is on its way already.

As for the runtime safety verification, I would love to see something that used proof-carrying code. For those who don't know about this, it is a way to guarantee runtime safety in the way that traditional sandbox-like VMs do at runtime, but statically, so that runtime overhead is eliminated and native machine-code can be run directly with all the same safety guarantees.3



All of these tools are non-language-specific, and they would help make creating new programming languages economically viable for software developers and startups in particular. In the end, this would advance the field of programming languages immensely, more than the development of any single language could.

---

1. I mean "language" in the most general sense, as it's sometimes suitable to abstract something away into a library and have your language be function calls into that library. Other times it makes sense to have a full-blown Turing-complete language. Anyway, this is why PL design is so important — it's really being done everywhere even though most people don't realize it.

2. What's missing from this list?

3. Peter Lee, a professor I worked for in college, was one of the inventors of proof-carrying code, and it turns out something so amazingly useful is practically a special case of type checking. However, this also requires proof-generation, another use of a good theorem prover. In the sci-fi stories that talk about computers programming themselves — they're talking about theorem provers. ...It's sad for us CSers. Most people are simply unaware of the amazing things currently in development.

Design — A Life Philosophy

Every problem is a design problem full of design choices. Even the choice to take a hands-off approach is a design choice. Every decision you make in your life — what to eat for lunch, what to do as a profession, who to hang out with on Saturday night — is a design decision. Each and every one of us does our best to design our life. Why would you ever choose something unless you were convinced that it would further you towards your idea of a successful life, for whatever your idea of a successful life is.

Joel gives a nice description of what design is in his 3-parter. Design is all about making trade-offs — as is life. Everything has an opportunity cost.

On Practicality and Tool-Making

A comment on my last post on design and programming languages spawned the following.

Q: Are you implementing some of your ideas?

A: It's always in the back of my mind to design and implement my dream programming language. This blog is in part my outlet for ideas that come streaming into my mind about programming languages. In fact, it's my only outlet. The more specialized you get, the fewer and fewer people you can find who consciously value your field; most people don't even know it exists. Consequently, I would just love to have someone to bounce ideas between who actually has a clue of what I'm talking about.

That said, I'm not actively working on something at the moment, and I think that warrants an explanation.

Back when I started programming, I primarily liked to make things that were useful on a day-to-day basis. In making such things, I started to see patterns and got the idea of making tools to make creating them easier and quicker. That eventually led me to programming languages, the medium of expression of all software. As a result, I spent practically my entire undergraduate career studying type-theory and compilers.

To this day, I am more convinced than ever that PL design is one of the most important things to {software as we know it}.

However, given all that, tool-making can be a huge side-track. Because like all things, there is a pattern of pattern-discovery and tool-making.

1. You use the tools you have to solve a problem.
2. You discover patterns in the use of those tools.
3. You make new tools that eliminate those patterns.
4. You return to step 1 with your new tools.

But if you're like me and see patterns everywhere, you could end up spending most of your time in step 3, not step 1. Especially when you see patterns in the tool-making process and start to recurse on sub-goals. And if your ultimate goal is to solve your original problem, there's something wrong.

The bottom line is that hard work — sometimes an obscene amount of it — pays off if it's focused on your goal. And the whole tool-making branch only makes sense towards your bottom-line if {the amount of work spent making your tool} plus {the amount of work it takes to accomplish your goal with the tool} is less than {the amount of work it would take without the tool}. This simple accounting formula has told me that making compilers and new programming languages is just not worth it given my goals. The cost of making a compiler and all the tools necessary to be productive with a new language is just so high that it dominates the inequality.

This can be frustrating, especially for someone who wants to create something beautiful.

But it is a matter of practicality.

Now, this may seem like an elaborate excuse to not work on something hard and important. But to me, it is the reason why I should work on other hard and important things.

I'm not trying to discourage anyone from working on compilers or designing new programming languages. On the contrary. If your goal is to make the best programming language in the history of programming languages — a so-called hundred-year language — then by all means, work on that! Seriously, please do; we are all in need.

All I'm really saying is that everything has an opportunity cost, and you should evaluate everything in terms of your goal, whatever that may be. Sometimes that means putting aside your own desires, like the desire to make a tool. But in the end, it's worth it.

1, 2, n

It's the 1, 2, n rule in software development.

You do something once. You do something twice. And the next time you do it, you generalize it for all times henceforth.

Waiting longer to generalize is painful at best. In the worst case, it leads to maintenance nightmares as when the same thing is done 14 times in 14 different places, each time slightly different than the rest.

Generalizing too soon however leads to other problems. Everyone's done this at one point in their coding career or another. Hey, this is a great idea! Why hasn't someone else done it before? I'll code it completely general the first time around so that it'll be done right, from the start. And then a few weeks of coding go by, you start to actually use it, and you find out the hard way that either it was a really bad idea or the API is so terrible that you have to re-write it. Because basically, you didn't iterate your design.

Code is all about design. And good design is re-design.

This is a big part of the reason why programming languages are so far behind. The amount of work that it currently takes to create (a compiler for) a programming language is so high, that the time it takes for a language designer to get feedback on his design is years. But by that time (if he was lucky enough to actually get users), he's already invested so much into it that he is unlikely to throw it away and start from scratch — something a good designer must do. On top of that, even if he wanted to start from scratch, it would take many months to de-program his mind from thinking in the previous language. (Like the first time a Java programmer learns Haskell and tries to write a loop!)

An IDE is part of a language. Error messages are part of a language. Interacting with other systems, written in other languages, is part of a language. They're all part of the interface of the language. And the interface is everything.

It's a chicken-vs.-egg problem though. Because to get all these tools made, based on history, requires that a language be popular. But how do you get people to use a language if it doesn't have the most basic tools like a debugger?

JavaScript is doing it right now. JavaScript tools — which make using the language no longer so painful — are cropping up everywhere these days. And the way it did that was by being the only choice if developers want their code to be executed in a browser window. Most languages aren't so lucky. So what is a language designer to do?

As a designer, I can't simply make a command-line compiler for my language. I have a vision of what editing code is like. And it's not with a text-editor. The closest thing I've ever seen to a true code-editor is Eclipse for editing Java code, but Eclipse has been around long enough for us to learn a few lessons. I can already hear the Emacs junkies crying about how Emacs is the ultimate editor. But again, it's still a glorified text-editor.

A true code-editor must absolutely be aware of the semantics of the language you are editing. It is inseparable from the compiler. Because the compiler knows how to interpret your code, but when something is amiss, it must be able to feed back to the programmer.

Writing code is all about design. And a designer needs feedback — as much feedback as possible. The quicker he gets that feedback the better, because he can make any necessary changes to the design only after he's gotten feedback on his current design.

So any code-editor that doesn't give immediate feedback on parse errors and requires that you do a full build is simply crap! How can you possibly work like that?! Ditto for type errors, link errors, and test-assertion errors. People see it as obvious when the old-timers tell stories about how they used to submit compilation jobs to mainframes on punch-cards, only to find out the next day that they forgot a semicolon and had to re-submit the entire job. Yet when it comes to where we are, here and now, people are blind.

Frankly, I'm getting tired of it. I want to see test data flow through my code as I write it and the unit-test results change from red to green as I fix a bug. I want to be able to highlight two blocks of code and have my code-editor factor out a function, replacing the two blocks with applications. I want bottlenecks to be highlighted in-editor based on profiling results. And compilation should be done completely in the background so I don't have to wait after making a change to use it. And this read-eval-print-loop that's all the rage nowadays might actually be useful if I could use it in the context of a particular point of my code.

Is it really that hard? Someone please tell me I'm a visionary because I can't believe that this can't be done. Nothing a little universal substitution and computing power can't solve.

Quantity Implies Quality

Quantity implies quality.

Example: In recent news, a professor searched 2,985,984 possible candidates for a universal Turing machine. In an earlier era when computing power was scarce, that may have been too many to search in a practical amount of time. But this guy found the answer. (Well, maybe.)

Example: Chess-playing is currently a prediction problem, but if resources increase enough to search the entire set of possible moves (which is theoretically possible as far as I understand the rules), winning becomes a search problem — which can always be solved. Think of tic-tac-toe. Or even checkers. In some games, a player with an abundance of computational power can look ahead to see which moves will lead to a win. Or at least a tie.

...I know what you're thinking. Wait, chess-playing is still a prediction problem. As with checkers, perfect look-ahead will tell you which moves will lead to the highest number of winning game-states compared to the total number of states. But your opponent, if he's smart enough, could always choose one of the ones that isn't a win for you. So it is more about predicting which moves your opponent will take. But I think this takes chess back to its roots — the fundamentals of a game.


The point is... some problems which are fundamentally search problems get transformed into prediction problems because the search-space is too large to be searched with current resources in a practical amount of time. Thus, the potential of a final right answer is lost.

By increasing resources to an abundance and using the law of Universal Substitution, you can re-convert problems back to search problems.

Quantity really does make a difference.

Everything Is a Compiler

All algorithms can be classified into one of the following.

• Search Engines
• Prediction Engines
• Networks
• Operating Systems
• Compilers

This becomes obvious when you look at what programs do from a high level: search, predict, communicate, manage resources, and translate/transform.1

Search

Searching is when you start with a question and your goal is to determine the answer. The answer may be fuzzy, but there is a definite best answer that can be tested. In other words, given two potential answers, a search engine has the ability to test which answer is the better one. Moreover, there are a finite number of potential answers. Examples include string searching and graph shortest path algorithms.

Prediction

Predicting, like searching, is when you start with a question and your goal is to find a good answer. Unlike search, there are potentially infinite possible answers and/or the best answer can not be tested explicitly. Heuristics must inevitably be used, but there is no guarantee that the chosen heuristics will lead to a good answer. Like fitness functions in a genetic algorithm (which falls under the prediction category), local maximas can be hit without indication. Current chess-playing algorithms are also prediction engines since they can not search the entire game-space of moves and must guess which move will more likely lead to a win.

Networks

Networks are made up of multiple entities that are independent in some respects, and the goal is to transfer/transport resources between those entities. In the case when the resource is information, the goal is to communicate. Any problem where entities do not have perfect knowledge of the entire system can be a networking problem. For example, a P2P search engine must use a networking algorithm or protocol to determine the existence of a target on a foreign entity, assuming the searching entity does not have an index which would amount to all-encompassing (possibly perfect) knowledge.

Operating Systems

Operating systems manage limited resources. When a resource is abundant compared to demands for it, management of the resource is not needed. When it is too rare to meet demands, management can not help. But when a resource is scarce, care must be taken to ensure that it is not abused, especially when entities with differing goals are competing for the resource. This is the goal of operating systems: to prevent abuse of scarce resources.

Compilers

Compilers take an input and give an output that somehow corresponds to the input. In the case of a programming language, a compiler takes in a program in a source language and outputs a program in a target language whose semantics are the same as that of the input program. In this way, the input and output correspond. However, the correspondence need not be unique (or surjective or injective). The compiler acts as a map from one language to another. Moreover, the language need not be a programming language, but something more general like a GUI. In this context, the languages that a compiler translates between are languages in the most general sense — any symbols. If the symbols of the language represent something and have meaning, then we usually call this translation; if they don't, we call it transformation.

Analysis

How is compilation different from search?

Fundamentally, both search engines and compilers transform an input into a corresponding output.2 The difference is in their approach. With search, there is a finite answer space and the answer is determined via exhaustive comparison with the input, whether that comparison is done at query-time or indexing-time. With compilation, the answer is determined via formulaic transformation of the input. There is a formula or method to get from the source to the target without ever touching all possibilities.

How Is This Helpful?

Realizing that a program you are making is implementing a certain kind of algorithm allows you to take advantage of all previous work on algorithms of that type. Currently, the way primitives and libraries are organized in most languages, they don't lend to revealing the high-level purpose of a program, and it's actually very easy to slowly go from a few parameters   to a configuration file   to a domain-specific language without even realizing it.

When things become that complex, you will wish you had designed your configuration interpreter as a compiler, which people have been making for years   and in general have well-designed methods of handling issues that come up during compilation. Likewise for searching and network protocols. And I suspect the same is true for prediction and resource management.

Algorithm Mismatches

Some algorithms pretend to be in one category when the problem they are trying to solve is fundamentally in another category. For example, web search. Certain cases aren't really search at all; they're prediction — because a person doesn't know what he's searching for until he's seen it. In other words, a person types in "Brad Pitt's birthday" as a Google Search, and there is a definite answer. But if someone searches for "cook", does he mean "cook" the verb or "cook" the proper name? If he means the name, does he mean the company, the county, or the person? (This is exactly why there are things like Clusty.) Or maybe his intent was literally to find all the things related to the word "cook", which most search engines assume. Because this is a search problem they can solve.

But the fundamental problem is ultimately prediction because even if the search weren't ambiguous, say it were about cooking the verb, what about cooking is the person looking for? Maybe the person is looking for a good recipe to impress his honey this weekend. Or maybe he doesn't know how to cook at all and wants to learn. Or maybe he's a master chef wanting to read articles about cooking.

An application developer will say that if this were the case, the person should have typed in exactly that: "articles about cooking". But my point is that even if he did type in "articles about cooking", which article is the one that will satisfy him? And this is something the person does not even know, until he's seen it. This is why the ultimate problem is prediction, not search. Search is merely an approximation. It may be an extremely useful one, but it's still an approximation. And seeing this mismatch will allow you to take your algorithm to the next level.3

Implications

Repetitiveness of Programming

Programming — as often as programmers attempt to never repeat themselves — is an extremely repetitive task. From a high-level view, all programs do these 5 things in varying combinations and details of problem domains. So the question arises... Can we simply master these 5 things, create the ultimate library for each of them, and only specify the details or parameters when writing a new program? At first, this seems like what we do already with programming languages. However, what this pattern suggests is that the primitives of the language should be Searching, Predicting, Networking, Managing Resources, and Compiling. Not looping, maintaining data structures, or other more low-level operations that are inherent in implementing the Big 5.

This is not a brand-new idea; it is just taking the next step. Already we see other languages that have just one of these primitives, and they excel beyond the rest. For example, Perl's regular expressions are a form of searching. They did it right and made it a language primitive. When people saw how powerful Perl regular expressions were, they duplicated them in practically every other language. Regular expressions though are still very specific in that they are only designed to search strings for a limited kind of patterns.

What would the interface to these 5 modules look like? I'm not sure, but the ultimate goal is that they entail a much higher-level language than mainstream programming languages today.

Compiler-Compilers

When you start to look at what I'm suggesting, you might recognize that this has already been thought of. It is the idea that your program is a specification for a search engine or a specification of a compiler, and your new compiler knows how to create the search engine or compiler from your specification of it. In other words, it is a search engine compiler. Or a compiler-compiler.

Years ago, much research was done on creating compiler-compilers. As a result, we have tools like Lex and Yacc. "Yacc" even stands for "Yet Another Compiler-Compiler". The research went further though, attempting to automatically generate the next phase of compilation. Other phases have been completely automated in a similar way with tools like ml-burg (or its equivalent in other languages like JBurg).

Similarly, a specification of a network protocol should entail its implementation. A policy for resource usage should entail its implementation. Likewise for the other categories.

Then the specification to combine these — yet another instance of one of the above types — yields the whole program.

---

1. If you haven't noticed by now, I don't usually bother with proofs. All axioms and even the rules of inference themselves are intuitive assumptions. For those who don't see that, I challenge you to try to prove induction, or better yet   deduction. (My study of epistemology taught me that intuition is the complementary tool of logic, and that the only way to know something for certain is to know the knower.) I realize this opens me up to a lot of criticism, but so be it. ...As long as we keep questioning.

Are there any categories that I missed?

I didn't include databases even though some might argue they are more fundamental since all algorithms use some kind of data structure. However, even though data drives all algorithms and the algorithms themselves are a form of data, no one would care about the data if they could get the results of the algorithms without the data. So databases are an implementation detail, and I am only concerned with the highest level in this post.

2. If you want to get really high-level, every program does this — transform input into a corresponding output. And this is what led me to the title "Everything Is a Compiler". ...Except perhaps truly intelligent machines, which some argue   are more than I/O.

3. Realizing that web searching is really prediction, you start to see that current criteria like frequency of keywords on a page or page rank make up the fitness function of a genetic algorithm. The fittest pages are then presented to the user in search results, who then clicks through and/or refines his search query. Smart search engines actually learn from this user-interaction — hence things like Google's search history and bookmarks. And they evolve their fitness function based on this. Voilà — co-evolution drives one of the most used, most influential websites of all time.

Perl Is Weirdly Interesting

I'm learning Perl now. I've used it before but never got deep into it.

From a logic-foundations perspective of programming languages, Perl makes absolutely no sense. Its semantics have no rhyme or reason. It is completely illogical.

I searched for a Perl language spec and found this on wikipedia.

There is no written specification or standard for the Perl language, and no plans to create one for the current version of Perl. There has only ever been one implementation of the interpreter. That interpreter, together with its functional tests, stands as a de facto specification of the language.

This basically says to me that Perl was designed as much as English was designed. And indeed, Perl is compared to a natural language.

I searched further and found the Perl language reference. And the Perl syntax page describes semantics, not syntax at all. On that page, there are numerous occasions that explain a rule, explain how there are exceptions to the rule, and then warning you to "don't do that"! It points out other things you "probably shouldn't rely upon".

...But that's just it isn't it. Perl is illogical. But it is intuitive.

Its intuitive semantics can be seen in conditionals. The following is from the Perl syntax page.

The number 0, the strings '0' and '', the empty list (), and undef are all false in a boolean context. All other values are true. Negation of a true value by ! or not returns a special false value. When evaluated as a string it is treated as '', but as a number, it is treated as 0.

What??! ...The only types that seem to exist in the core language are scalars, lists, hashes, and functions of arbitrary signature. References seem to be the only thing with an arrow kind.

All of this starts to make sense when you read about how Perl is an analogy to natural languages. Most languages look at programs as machines. Perl looks at programs as novels. There's nothing wrong with either way; they're just different. In fact, this is exactly what makes Perl interesting.

The design of Perl seems to be that of a medium not for engineers, but for people who create works of art or works of literature.

One claimed benefit of its natural-language-ness is that you can learn as you go as there are many levels of acceptable competence. (Take my competence in English for example, using words like "natural-language-ness".) This results in there being many poorly-written Perl programs, and Perl advocates argue that the burden in creating well-written code is rightly put on the programmer, not the language designer. The alternative however, is that those incapable of writing beautiful code never even learn the language. Whether fortunately or not, a language will always be judged to a large extent by the programs written in it.

Dabbling in Haskell recently, I am definitely going to learn more Perl simply for the contrast, as trying different things leads to more patterns.

Methods to the Aha

When I was a fresh CS major at CMU, I took the standard 15-251 Great Theoretical Ideas of Computer Science class. The professor Steven Rudich introduced us to some amazing topics. Even though I was never able to use it to its full potential, the most valuable thing taught was what he called the "aha! method". In other words, he taught that there was a method to getting to the point where a light bulb goes off in your head and you get an insight that allows you to solve a hard problem, making you say "aha!". And moreover, that this method can be learned.

The list he gave was as follows.

• Phrase Hygiene
• Representation
• Induction
• Modularity
• Exemplification
• Refinement
• Abstraction
• Bracketing

Briefly, he meant: tag assertions with phrases that indicate your degree of conviction; actively choose your representation of the problem; use the various forms of induction; break the problem into smaller subproblems; use small examples; don't give up after you've found an answer; abstract away the non-essential details of the problem; try to give upper and lower bounds and make them converge. I'm not going to go in to more details of each of them, but the overall idea is that each is a trick to help you more easily see the pattern in the problem that will lead you to a solution.1

As Rudich pointed out, in any field — martial arts, violin, tennis, magic, programming — the novice makes a huge motion, the black belt makes a small motion, and the master makes a tiny motion. The more complete the pattern you see, the less work is required to accomplish a task.

Recently, I've found a new method. What made me see this was an "interview question" I heard way back when I was in Rudich's class. It goes something like this... You have to cross a gorge that's 50 feet wide and 50 feet deep. All you have is two 20-foot ladders and all the rope you need. How do you do it?

And no, you can't go around it. What now? (You might want to actually think about this before reading on.)

Select text here for the spoiler... The answer is — since you have all the rope you need — that you fill the gorge with rope, and walk across.

Now, the answer to these riddles is often obvious in hindsight. But what can we learn from it? Why couldn't we see the answer before? It wasn't until recently that I realized that if you abstract away the details of this solution, what you had to do was substitute an abundance of one resource for the lack of another resource. And this I've discovered   is another method to the "aha!": Universal Substitution.

What I mean is that any resource can be substituted for any other resource. And this is simply amazing!

...As I mentioned in my last post, analogy seems to be a pattern of all of these. The Representation method above seems to hint at that. But how do you know which representation to choose? You have to already have made a connection — an analogy — with another problem.

Jeff Hawkins seems to think that all problems are solved through either memory or analogy. In other words, you either remember the solution to the problem if you've seen it before. Or you relate the problem to another problem and remember the solution to that, and then apply that solution to the specific situation at hand   which is something your mind does all the time.2

If this is true (which seems to make sense), then analogy is a problem-solving method. But not only that, it is a method to finding problem-solving methods! It is a method of solving any problem, even meta-problems and meta-meta-problems.3

So this leads us to our next question. How do you become good at creating analogies?

Analogies are connections between two different things. They are the result of abstracting something away from two things that are different and being left with things that are the same. Analogies are patterns.

One thing I've noticed that increases the likelihood of seeing patterns is trying different things — all sorts of things. Oftentimes you suddenly realize that things you've become so used to don't have to be the way they are. That it is not necessarily the right way or the wrong way. But also, when it's different enough, you can't help but see what isn't different. In other words, what remains constant — which is a pattern.

You never know where a pattern might show up, or what lines it might cross.4 Plus, oftentimes the analogies within a field have already been exhausted by others, while cross-discipline analogies lay un-reaped.

Also, there is the trap of becoming too close to something. After working on a problem arduously for a while, it is often helpful to first take a break, and then step back and look at the problem to get a bigger perspective. In other words, disentangle yourself from the details and take a look a the whole picture at once, just as if you were putting together a jigsaw puzzle.

All this really is, is another example of the law of Balance.

I've actually attempted to devise an algorithm for finding patterns before. The thing is, there are a seemingly infinite number of ways to categorize things. So if you try to categorize things and then look for something re-occurring, you'll never know if you're simply using the wrong categorization. That's why it's hard to learn something new in general. According to Jeff Hawkins, you have to see a pattern before you can learn anything. The people who tend to be seen as smart are the ones who pick up new things quickly, and they can do this because they see the patterns quicker than others. But it is a completely unconscious activity to them.

This begs the question though, is there another way of finding patterns? Can we find patterns without first categorizing things?

It's like if you have a hash function that abstracts away from things you're looking at, and you keep hashing different things hoping to find a collision. Once you do, you've found a pattern. For example, if your hash function took physical objects as input and gave a color as output, you could hash all the things on your desk and if any items resulted in the same color, you would have found a commonality between those two things, which is a simple yet significant pattern.

But what if you're looking for another pattern? Which hash function should you use? Is there another way besides this hashing method?


See also: Side-Stepping Obstacles in Space and Time

---

1. Try these out on this logic puzzle; they really work. The method can take you from being completely stuck to a point where you can work through the problem.

2. What Jeff Hawkins describes is abstraction and application. Sounds like functional programming to me. Lambda abstraction and function application. Is there anything more fundamental?

3. This really makes you wonder about things like Numenta, Jeff Hawkins' implementation of the brain which he calls hierarchical temporal memory. What problems will be solved once the size of a brain becomes limited by the amount of money someone is willing to throw at it, instead of by the human skull?

4. I'm not at all interested in sports, but recently I went to a baseball game with my brother, and I realized that anything — once you start optimizing, it becomes something so deep that you could spend an entire lifetime mastering it. Baseball. Cooking. Driving. Conversing. ...What then, is truly worth mastering?

Programming Is Writing

Programming is just like writing. Prose that is.

Specifically, I'm talking about essays — prose in which the purpose is to explain something. Which is exactly what a program does. It explains an algorithm.

Both programming and writing are constructive, as in the constructive logic sense. Writing a program is writing a proof, is writing an essay. In both, it is desirable to be clear and concise. What you write must be organized; the most straight-forward way is starting with a high-level overview, or a top-down organization. Both programs and prose must be read and understood sequentially. Sure you can jump around, but in both, each statement must follow from the previous one. And sub-points, or sub-goals, must flow from beginning to end to form the final point.

In prose, in addition to giving explanations, you give examples to get your point across. So why don't programming languages allow you to write examples as part of the source? In a sense, pattern-matching is exactly this but very limited. Is it not done simply because compilers are not good at inferring from examples? Even if this is the case, we shouldn't let it stop us. The purpose of a programming language is to allow an algorithm to be expressed, and secondarily to be executed. Now I know the economists out there will argue with me. And you're right. But I'm talking more from a design perspective.

Stories are also proofs — they are illustrations of {proof through action}. That's why theories often start from anecdotal evidence. Anecdotes are short story proofs. Everyone uses them all the time. Story is the universal proof-language. In other words, the universal programming language.

Now, I wouldn't want to write all my programs in this universal language for practical reasons. English for example, is not optimized for expressing algorithms. But it's universal in the sense that anyone can understand and relate to stories in the language — i.e. programs in the language — because they describe life experiences, which English is optimized for.

I was talking with Kyle the other day about how I and other programmers sometimes alienate people when we speak because we either have to clarify intuitively obvious ambiguity   or explain things by decomposing them into such detail that the explanation becomes incomprehensible to others. The other day, someone asked Kyle if the hat he was wearing was new. He replied by asking what the person meant by "new". From an intuitive perspective — the perspective of the person asking the question — Kyle should have answered simply "yes" or "no". But instead, the inquirer became slightly confused by the unexpected response, and they went back and forth going nowhere.

Kyle claims that programmers are better at recognizing puns because puns are instances of ambiguity, which programmers have been trained to spot. (I disagree though, because I find myself missing obvious puns, and I think this has to do with the seemingly under-developed right hemisphere of my brain, or at least the disconnectedness of the two hemispheres.) But that got me thinking... if programmers are good at spotting puns because puns are instances of ambiguity, wouldn't that make them bad at creating puns? Because I read recently in On Intelligence that part of what makes Shakespeare a genius in writing is how he can create a metaphor between two things and not actually be talking about either of them. I'm not an expert on Shakespeare, but I do know it's hard to use metaphors in writing. It's simply more abstract.

So if programming is just like writing, what would be the equivalent of a pun in programming?

A pun isn't merely ambiguity between two or more different meanings; it's actually double meaning. Triple meaning. Any or all of the meanings make sense.

So to have this in programming, for example, when a language has two namespaces: one for variables, and one for types; you would have to declare a name in both namespaces and find a way to state the name in such a way that both the variable and the type make sense.

Are there any languages out there that allow for such a thing? I'm not sure. But an obvious case of this would be for printing in a language that has type information available at runtime. You could write something like print foo where foo is defined as both a variable and a type. Which should it print? Obviously it would depend on the semantics of the language. But my point is, it could do both. Or it could look at you and smile coyly.

Like puns and the idea of including examples directly in the source, not separately in test cases, I believe there are more good ideas to be extracted from this analogy that programming is just like writing. Can anyone out there see any more? I'm working on a post about problem-solving methods that I hope to finish soon, but analogy seems to be an amazing problem-solving technique, if not a pattern of problem-solving techniques in itself. In other words, a meta-meta-solution.

Of Little Importance

I've been going crazy writing recently. I've got 7 other posts that I started this week that I haven't had a chance to finish. The reason I've been having so many ideas for posts is partly because I've been seeing so many patterns. So why have I been seeing so many patterns about programming?

I went to hang out with my crazy-cool cousins this weekend. The kind of cousins I only see a couple times a year, but every time it's a blast. And at some point I realized from a conversation that what I'm doing with programming — becoming a relative expert on it, focusing all my time on it, reading about it, writing about it — other people do all the time with sports, video games, movies, and other things. In other words, it's no different. No better. No worse. And the more I focus on programming, the more I will prosper at it. ...But at the same time, I won't be prospering at what's important in life.

This is the pattern of wasting your life, getting caught up in something with only relative value.

I recently saw a documentary called Flight from Death: The Quest for Immortality, and it made me realize that instead of trying to live a great life, if you live your life to make death acceptable, you will end up with a life greater than you can possibly imagine. And the phrase "it's a good day to live" will become synonymous with "it's a good day to die".

I don't believe it's chance that the times I've been on the edge of death were also the times I've felt most alive. It was a coincidence — a coinciding of events. In other words, a pattern. This also points to degrees of being alive — i.e. a life/death continuum — rather than the common conception of a binary switch.

So let's remember to focus on what's important.

PL Feature Wish List

Here's my programming language feature wish list:

• Parallelism using list comprehensions like nesl
• Seamless parallelism over a network like erlang, with ability to extend with encryption/security
• Turing-complete type-system like Qi, but with implicit runtime verification
• Turing-complete macros like Lisp (i.e. extending the compiler), but maybe just the ability to have function parameters optionally evaluated which would give you the equivalent, assuming all computations that can be evaluated at compile-time are
• Expose the interpreter/compiler for meta-circular evaluation
• Arbitrary meta-data like Java annotations or .NET attributes on all source code nodes, not just functions.
• Full source reflection at runtime, compile-time, anytime (although this might not be necessary if the IDE has scripting — see below)
• In other words, blur distinction of compile/run/etc. times
• Bi-directional mappings for implicit parsing and printing (i.e. syntax skinning) with ability to use them inline (which would, among other things, give you the usefulness of reader macros)
• Seamless FFI and importing of Java libraries — we have to build on top of something so it might as well be the thing with the most well-developed libraries to make the language a viable option for production code
• Seamless interaction with the native platform when needed
• Infinite data structures like Haskell — I guess this means lazy evaluation
• Seamless language-integrated persistence
• Type inference with useful error messages when my programs fail to type-check
• Namespaces — I've never seen a language sustain large applications without these, and many languages have failed to catch on
• Modules and true functors (i.e. functions over modules)
• Type-classes — they're really useful
• User-definable monads
• Examples/test cases in source

An IDE with:

• Searching by function type-signature, which will also return functions that can be used with simple coercion functions on the parameters (e.g. using stringOfExp : exp -> string as an implicit coercion)
• Graphing of module and function dependencies so I can see where the entry-points are when I am trying to look for something in a module I'm not familiar with
• Profiling which feeds back into the editor (color-coded perhaps) so I can see where I should optimize when I'm editing
• Scripting like Emacs but in the language that it edits, not a variant of Lisp or some other random language that the editor happens to be written in
• Incremental background compilation with automatic dependency analysis
• Knowledge of the abstract syntax tree to allow for (scripted) refactoring. In fact, why not define parts of the language (or skin) as mappings between UI events and semantics, taking the idea of bridging the gap between abstract syntax trees and the user even further.
• Type inference that feeds back into the editor, as if you had typed it in
• A graphical window designer like Visual Studio's Form Builder or Xcode's Interface Builder
• Toggling of debug output which is structured, not just text, so that more information can be expanded without re-running. Toggling should be optionally automatic by tracking which pieces of code have changed recently and only showing dataflow affected by the changes.

A library with:

• Good data-structures (e.g. hash tables, balanced trees, heaps, graphs, etc.)
• Hash functions on common types and typed encryption algorithms (i.e. an 'a rsa_encrypted type) with an interface that makes them easy to use (i.e. no coding cost and little knowledge of encryption algorithms required)
• A hardware/platform abstraction layer for every-day drawing using a box-model similar to CSS

Notice that just about everything here is agnostic to what the actual expressions of the language are.

Is this technically feasible? Absolutely. Just about all of it's been done before in one form or another.

Is this realistic? No. But no one ever made it anywhere interesting without dreaming big.

I'll add to this post's list when I think of something else.

Why Tagging Doesn't Work

It requires refactoring.

For example, when I bookmark something on delicious, I tag it with all the things I think are relevant to that site at the time. But the way I categorize sites and what things I believe to be important   change over time. So the only way I can search through my bookmarks to find all the things related to a certain topic that I wasn't even aware of before   is to linearly scan through all my bookmarks and re-tag them.

Is there a solution to this?

The Mind Is Strict

Don't think of a red ball. □
:-P

Birthday Patterns

Here's a cool experiment to try. Don't read ahead without actually doing this though or you'll ruin the results.

Choose a few people and categorize them. In one group, those who you consider to have a "down-to-earth look", and in the other group, those who you consider to be "hot". That's right. You must choose one. No one can be in both categories.

Who should you choose? Anyone you can find out their birthday or astrological sign easily. But it helps prevent bias if you don't already know their birthday. So use people you don't know their birthday but can easily find out, like a friend on Facebook or a celebrity.

For example, Jennifer Aniston: down-to-earth; Britney Spears: hot.

Once you've categorized the people in your list, look up their birthdays. Notice anything? A pattern?

If you don't see anything, try categorizing people whose birthday is in the first half of the year versus the second half of the year. Now do you see a pattern?

I've found this is scary-accurate. Almost to the point where it's a little creepy. I was playing this game with my friend who saw the pattern originally where we would pick a celebrity, guess if they were born in the first half of the year or the second based on how they look, and then look up their birthday. We found it worked much better for females, but this is probably due to the fact that we've been training for years judging females on looks. I suspect though that it works just as well for everyone as long as you have a trained eye.

When you start to think about this, the implications are crazy. Behavior and beauty — and as my friend theorizes, how suitable someone is as a spouse — are determined with high probability by your date of birth. I'd like to see skeptics of astrology try to explain this away.

I'm not very knowledgeable about astrology at all. I'm just beginning to notice that there really is some truth to it. We stumbled upon this little game last night after I looked at the birthdays list of my friends on Facebook. I noticed there was a disproportionately large number of birthdays in certain months. So I decided to count them up and graph them in Excel.

Wow! The results were stunning. Approximately double the median of birthdays per month appeared in a single month for my friends. The next highest month wasn't even close. What does that say about me? ...Apparently I've been inadvertently choosing friends based on birthday. Or your birthday really does correlate with your personality, of which I have a certain preference.

I know the skeptics still won't buy it. But that's okay. Those who see will prosper anyway. Because this is not just evidence of the pre-determination of your life, but also of the rule of Balance. There are rules to the Universe — patterns — and one of them is that you can't be both smart and beautiful, unless of course you're missing something else!

Static Vs. Dynamic Typing

Static vs. dynamic typing. Here's my ravings.

Being thoroughly trained in ML, I understand the benefits of static typing. Here are some claimed benefits.

1. Reduced CPU usage since types don't need to be checked at runtime
2. Reduced memory usage since types don't need to be stored at runtime
3. Other type-directed optimizations
4. Discovery of errors as early as possible, for all my code
5. Static typing is more powerful/expressive since dynamic typing is a special case, or subset, of static typing where everything is tagged

Now let's look at the claimed benefits of dynamic typing.

1. Easier to run and test since there are practically no compile-time or link-time errors
2. Programs are not limited by the expressiveness of the type system of the language — e.g. heterogeneous data structures w/o explicit tagging
3. Allows for implicit open recursion, late binding, dynamic scope, and duck typing
4. Programs are simpler since I don't have to worry about types. Things are often coerced implicitly making my programs more concise, and I don't have to use really abstract things like type variables and higher-order types. In other words, they have been abstracted away by being pushed down to a place I don't have to think about.
5. Faster prototyping due to not having to write down types which change often during prototyping

There are a few claimed benefits, however, which are not true. Allow me to refute.

Myth: Static typing is more powerful/expressive since dynamic typing is a special case, or subset, of static typing where everything is tagged.

It's true that dynamic typing is a special case of static typing. This can be seen by simply creating a tagged union where each branch is a different possible runtime type. Then make every expression in your language a value of this type. Voila — dynamic typing.1

However, it does not follow that this makes static typing more powerful. To see this more clearly, let's use a simpler example. C vs. assembly. C is more powerful than assembly. But there are things you can write in assembly that you simply can not write in C. Because you are restricted. Restricted by the definition of the C language which does not know anything about your cool new machine instruction to factor large numbers and break your gramma's closely-guarded RSA-encrypted banana nut bread recipes. However, anything you can write in C, you can obviously write in assembly. So C is a special case, or subset, of assembly.

In other words, the more powerful language is the one that is the subset.

And this does not depend on the specifics of the language. So in the case of dynamic vs. static typing, dynamic typing is actually more powerful. I alluded to this already by the fact that dynamic typing abstracts away types by pushing them down into the runtime system.

Myth: Dynamic typing allows for faster prototyping due to not having to write down types which change often during prototyping.

This myth stems from the belief that types in dynamically typed languages don't have to be written down at all, while types always have to be written in statically typed languages. And this is absolutely false!

First of all, even though you don't have to declare the types of variables and functions in dynamically typed languages, you still write down some types. Whenever you make a class, declare fields or methods of a class, or specify inheritance relationships, you are creating and/or specifying types.

Secondly, type information does not need to be specified in statically typed languages that include type inference. Many languages with type inference like ML and Haskell are specifically designed so that you don't have to write down any types. The only exceptions are enumeration-like data structures (i.e. tagged unions) which have runtime values. So the difference between the amount of type information you must write depends greatly on the language itself, not whether that language has static or dynamic typing.

So we are down to the following for static typing.

1. Reduced CPU usage since types don't need to be checked at runtime
2. Reduced memory usage since types don't need to be stored at runtime
3. Other type-directed optimizations
4. Discovery of errors as early as possible, for all my code
5. Static typing is more powerful/expressive since dynamic typing is a special case, or subset, of static typing where everything is tagged

And for dynamic typing.

1. Easier to run and test since there are practically no compile-time or link-time errors
2. Programs are not limited by the expressiveness of the type system of the language — e.g. heterogeneous data structures w/o explicit tagging
3. Allows for implicit open recursion, late binding, dynamic scope, and duck typing
4. Programs are simpler since I don't have to worry about types. Things are often coerced implicitly making my programs more concise, and I don't have to use really abstract things like type variables and higher-order types. In other words, they have been abstracted away by being pushed down to a place I don't have to think about.
5. Faster prototyping due to not having to write down types which change often during prototyping

Some people have counterarguments as to why these are not real benefits. I will go through some of them.

Myth: The cost of runtime type information in speed or memory is not significant, especially with today's hardware.

In general keyboard-speed applications, the cost of runtime type information is not significant. But in core loops, it can be huge. Because it is not just a matter of checking the types, which itself can be very costly in a loop. But the fact that you must store everything as a tagged value means that everything must be stored as a pair in memory. This means that in dynamically typed languages, there is no way to specify the computation of adding the values in two registers and storing the result in another register. The add operation in a dynamically typed language must load a tag from memory, conditionally branch on the value of the tag, load a 2nd tag from memory, conditionally branch on the value of that tag, load the first number from memory, load the 2nd number from memory, add the numbers, store the result in memory, and then store a new tag. This is ignoring the case where the runtime system has to allocate memory for this new object, as the statically typed version may spill registers, so call it even.

For such a simple operation though, this is a huge cost.

Myth: Type-checking is a substitute for testing.

Again, absolutely not true. Although some people may use type-checking (static in particular) for testing program correctness, this is neither its intended purpose nor its limit of use. Type-checking is more about ensuring meaningful, well-defined execution than ensuring correctness. Even though you may hear strong static-typing advocates say, "When your program type-checks, you'll often find that it just works", this is simply not true for large, intricate programs. Although type-checking may help you find errors, it is not the same as testing. Thus, it is not a suitable substitute for testing.

Static type-checking will however discover one entire class (or a few classes) of errors at compile-time, early in the development cycle — something that is dependent upon executing every code path in a dynamically-typed language, which is simply infeasible.

Other Issues

Open Recursion, Late Binding, Dynamic Scope

Open recursion and late binding are just special cases of dynamic scope. It's not clear whether having these features is a good thing or not. They make it easy to introduce bugs when extending code in a way that was originally not intended (which happens fairly often). They can definitely be useful though since they allow you to do things that can't be done easily without the features. But as I explained earlier, more flexibility does not imply power. In fact it's the opposite, as long as the constraints are well-designed. Except in bottlenecks where optimization is crucial, Duff's device is a bad feature for a language to support. And although you can implement open recursion in a statically typed language, you can't implement dynamic scope in general (and be type-safe) without converting your entire program to use tagged values, which would be implementing the entirety of dynamic typing. So in this respect, dynamic typing is actually more flexible.

Some may argue that the language should not constrain the programmer. Let him decide whether or not to use it. After all, exceptions are a special case of dynamic scoping, and many statically-typed languages support this. The problem is that you will inevitably be using or extending someone else's code. And the person who wrote that code will inevitably make a judgment call that you consider to be poor, making the code non-obvious and unnecessarily complicated from your perspective — which can really be said about any powerful language feature, so maybe it's just me. But this one can make it especially difficult to reason about your programs. ...Again, it's not completely clear whether these features are good or bad.

Type-Directed Optimizations

If you care about performance, which you will for just about any production app, type-directed optimizations can help you a great deal. I won't go into the details of how they work, but the basic idea is that the compiler uses type information to optimize your code. Theoretically, you could have a compiler for a dynamically-typed language that performed the same optimizations. However, to do this at compile-time, it would have to collect the same information as a compiler for a statically-typed language. I've never seen a compiler able to do this in general, but such a hybrid is the future of typing.

An alternative is for the compiler to do the optimizations at run-time — i.e. just-in-time.

The problem with static typing is that there are always more type-directed optimizations that would be nice to do is always more type-expressiveness that would be nice to add, and so more and more complicated type systems emerge, oftentimes complicating a language with quirks such as ML's value restriction. On the other hand, if dynamic typing were used, the problem causing such complication would disappear. A simple solution is often the best.

Types in the Source Code

Dynamic-typing advocates claim that writing down types is a hindrance, especially during prototyping. This stems from the assumption that you have to do additional work to think of and change the types of the expressions you're dealing with.

I definitely understand the criticism of writing down types because I don't usually write down loop invariants, so I could see that as being "extra work". But types are no different. Both types and loop invariants are required to create the program, but not to execute the program.

For anyone who's become proficient in a language with a powerful type system, he knows that in order to even write a meaningful program, you have to use the correct types. And so when you're writing the program, you implicitly (and unconsciously for some) give a type to every single expression you write. So why not write down all this extra information that was used to create the program so that it can be used not only by the compiler, but by you when you come back to your code after a week. The same goes for invariants — loop invariants in particular — but I'll save that for another post on theorem provers!

Knowing what I know about theorem-proving and loop invariants, I believe writing down the types is the right thing. However, in a prototyping situation where the types are changing faster than you can type them, you shouldn't have to write them down. That's where type-inference comes in. But as the system begins to stabilize — as some parts which are getting pushed down in the hierarchy do — you should write the types down, because at that point you know what the types should be, and the extra information can be used by the compiler to check and optimize, and by those reading your code to understand your program's interfaces and semantics.

Typing of the Future

Here's the typing hybrid I see as the future, as the answer is never on either end of a spectrum, but a balance somewhere in the middle — a pattern of the Universe.

1. The flexibility to omit the vast majority of types, but consequently,
2. a significant amount of type-inference that uses
3. a powerful type system that is so expressive that it is itself Turing-complete, which requires
4. blurring the lines between compile-time and runtime to the point where the programmer can not always tell (and should not always care) whether static or dynamic typing is being used by the compiler.

And we will get there by pushing on.

---

1. This is where the "un(i)typed language" joke comes from. A language that is "untyped" really has exactly one type for all expressions.