Old READMEs: Atlast... make everything programmable!

I have been exploring embeddable languages for the last month or so. I have learned forth and some of its many many many variants [1] and while exploring one of the most obscure ones called Atlast [2], I found a very interesting README which I will quote liberally in this post.

Virtually every industry analyst agrees that open architecture is essential to the success of applications. And yet, even today, we write program after program that is closed--that its users cannot program--that admits of no extensions without our adding to its source code. If we believe intellectually, from a sound understanding of the economic incentives in the marketplace, that open systems are better, and have seen this belief confirmed repeatedly in the marketplace, then the only question that remains is why? Why not make every program an open program?

And this is indeed very important. Any app worth developing is probably worth developing in an extensible manner. After all, you are probably never going to figure out what the user really wants to do.

But why is not every program "open" in this manner?

Well, because it's HARD! Writing a closed program has traditionally been much less work at every stage of the development cycle: easier to design, less code to write, simpler documentation, and far fewer considerations in the test phase. In addition, closed products are believed to be less demanding of support, although I'll argue later that this assumption may be incorrect.

This is true, although it was much more true in 1991 when Atlast was released to the public domain. Nowadays the generalized adoption of more dynamic languages has made this much easier. After all, you can write a plugin system for your python app in perhaps ten lines of code.

However, there is a strong backlash from the pains of MSOffice scripting, which involved loveliness like function names that changed according to your locale, and basic as the language of choice.

On static languages this is harder, but there are technological solutions. For example, in KDE there is already a limited scriptability culture based on how easy it is to provide external DCOP interfaces (I suppose I should say DBUS nowadays). Also for KDE4 the Kross scripting framework promises language-neutral scripting.

Most programs start out as nonprogrammable, closed applications, then painfully claw their way to programmability through the introduction of a limited script or macro facility, succeeded by an increasingly comprehensive interpretive macro language which grows like topsy and without a coherent design as user demands upon it grow. Finally, perhaps, the program is outfitted with bindings to existing languages such as C.

Noone provides straight custom C bindings for applications now, right? But yes, I expect scripting is not the first thing that gets implemented.

An alternative to this is adopting a standard language as the macro language for a product. This approach has many attractions. First, choosing a standard language allows users to avail themselves of existing books and training resources to learn its basics. The developer of a dedicated macro language must create all this material from scratch. Second, an interpretive language, where all programs are represented in ASCII code, is inherently portable across computers and operating systems. Once the interpreter is gotten to work on a new system, all the programs it supports are pretty much guaranteed to work. Third, most existing languages have evolved to the point that most of the rough edges have been taken off their design. Extending an existing language along the lines laid down by its designers is much less likely to result in an incomprehensible disaster than growing an ad-hoc macro language feature by neat-o feature.
Unfortunately, interpreters are slow, slow, slow. A simple calculation of the number of instructions of overhead per instruction that furthers the execution of the program quickly demonstrates that no interpreter is suitable for serious computation. As long as the interpreter is deployed in the role of a macro language, this may not be a substantial consideration. However, as soon as applications try to do substantial computation, the overhead of an interpreter becomes a crushing burden, verging on intolerable. The obvious alternative is to provide a compiled language. But that, too, has its problems.

Interesting point here is that again, the speed of interpreters is not so big a deal nowadays. After all, LuaJIT is pretty fast for most uses.

Using a compiler as an extension language is not something I have run into but I could do some pretty things using, for example, TCC, Python and python-instant.

Now we get to ATLAST itself:

ATLAST is a toolkit that makes applications programmable. Deliberately designed to be easy to integrate both into existing programs and newly-developed ones, ATLAST provides any program that incorporates it most of the benefits of programmability with very little explicit effort on the part of the developer. Indeed, once you begin to "think ATLAST" as part of the design cycle, you'll probably find that the way you design and build programs changes substantially. I'm coming to think of ATLAST as the "monster that feeds on programs," because including it in a program tends to shrink the amount of special-purpose code that would otherwise have to be written while resulting in finished applications that are open, extensible, and more easily adapted to other operating environments such as the event driven paradigm.
The idea of a portable toolkit, integrated into a wide variety of products, all of which thereby share a common programming language seems obvious once you consider its advantages. It's surprising that such packages aren't commonplace in the industry. In fact, the only true antecedent to ATLAST I've encountered in my whole twisted path through this industry was the universal macro package developed in the mid 1970's by Kern Sibbald and Ben Cranston at the University of Maryland. That package, implemented on Univac mainframes, provided a common macro language shared by a wide variety of University of Maryland utilities, including a text editor, debugger, file dumper, and typesetting language. While ATLAST is entirely different in structure and operation from the Maryland package, which was an interpretive string language, the concept of a cross-product macro language and appreciation of the benefits to be had from such a package are directly traceable to those roots.

This concept was later adopted by Lua, Tcl, and you could use Python in this manner, although usually it's done the other way around. Which means that there is not so much sharing of extension languages as there could be.

And onto the conclusions:

Everything should be programmable. EVERYTHING! I have come to the conclusion that to write almost any program in a closed manner is a mistake that invites the expenditure of uncounted hours "enhancing" it over its life cycle. Further tweaks, "features," and "fixes" often result in a product so massive and incomprehensible that it becomes unlearnable, unmaintainable, and eventually unusable.


Far better to invest the effort up front to create a product flexible enough to be adapted at will, by its users, to their immediate needs. If the product is programmable in a portable, open form, user extensions can be exchanged, compared, reviewed by the product developer, and eventually incorporated into the mainstream of the product.

This prefigures the current FLOSS ecosystem, which is pretty impressive for a product that was mature in 1991, although of course there was already a community of EMACS hacking and similar niches.

It is far, far better to have thousands of creative users expanding the scope of one's product in ways the original developers didn't anticipate--in fact, working for the vendor without pay, than it is to have thousands of frustrated users writing up wish list requests that the vendor can comply with only by hiring people and paying them to try to accommodate the perceived needs of the users.

True if a little too cynical for my current taste ;-) Of course in FLOSS there is no such conflict between users and developers because the developers can usually just whine explain to the users the realities of free software development. But hey, a low barrier to entry is always a nice thing to have.

Open architecture and programmability not only benefits the user, not only makes a product better in the technical and marketing sense, but confers a direct economic advantage upon the vendor of such a product--one mirrored in a commensurate disadvantage to the vendor of a closed product.
The chief argument against programmability has been the extra investment needed to create open products. ATLAST provides a way of building open products in the same, or less, time than it takes to construct closed ones. Just as no C programmer in his right mind would sit down and write his own buffered file I/O package when a perfectly fine one was sitting in the library, why re-invent a macro language or other parameterisation and programming facility when there's one just sitting there that's as fast as native C code for all but the most absurd misapplications, takes less than 51K with every gew-gaw and optional feature at its command enabled all at once, is portable to any machine that supports C by simply recompiling a single file, and can be integrated into a typical application at a basic level in less than 15 minutes?

And then proceeds to throw a Forth variant at you ;-) Good concept, perhaps not the nicest language to use, although the choice is very understandable.

Am I proposing that every application suddenly look like FORTH? Of course not; no more than output from PostScript printers looks like PostScript, or applications that run on 80386 processors resemble 80386 assembly language. ATLAST is an intermediate language, seen only by those engaged in implementing and extending the product. Even then, ATLAST is a chameleon which, with properly defined words, can look like almost anything you like, even at the primitive level of the interpreter.
Again and again, I have been faced with design situations where I knew that I really needed programmability, but didn't have the time, the memory, or the fortitude to face the problem squarely and solve it the right way. Instead, I ended up creating a kludge that continued to burden me through time. This is just a higher level manifestation of the nightmares perpetrated by old-time programmers who didn't have access to a proper dynamic memory allocator or linked list package. Just because programmability is the magic smoke of computing doesn't mean we should be spooked by the ghost in the machine or hesitant to confer its power upon our customers.

Oh yes. Libraries rule, and when there is no such thing as the library you need, it's the worst position to be in.

Don't think of ATLAST as FORTH. Don't think of it as a language at all. The best way to think of ATLAST is as a library routine that gives you programmability, in the same sense other libraries provide file access, window management, or graphics facilities. The whole concept of "programmability in a can" is odd--it took me two years from the time I first thought about it until I really got my end effector around it and crushed it into submission.

I am probably going to use ATLAST or something similar in a program, but the user will not see a forth-like language at all, but a classical Excel-like formula language which would get compiled to the forthish language behind the scenes.

Open is better. ATLAST lets you build open programs in less time than you used to spend writing closed ones. Programs that inherit their open architecture from ATLAST will share, across the entire product line and among all hardware platforms that support it, a common, clean, and efficient means of user extensibility. The potential benefits of this are immense.

Indeed. Probably not using ATLAST, but Lua or Python or something else instead. And still, 16 years after ATLAST was released to the public domain, we are still walking down this road, and not even close to the goal.

[1] As they say, "Once you see a forth, you have seen a forth"
[2] The "Autodesk Threaded Language Application System Toolkit", by John Walker. Version 1.0 was released in August 1995, 1.1 in July 2002... I expect 1.2 around August 2009 :-)


Comments powered by Disqus