I have been ex­plor­ing em­bed­dable lan­guages for the last month or so. I have learned forth and some of its many many many vari­ants [1] and while ex­plor­ing one of the most ob­scure ones called At­last [2], I found a very in­ter­est­ing README which I will quote lib­er­al­ly in this post.

Vir­tu­al­ly ev­ery in­dus­try an­a­lyst agrees that open ar­chi­tec­ture is es­sen­tial to the suc­cess of ap­pli­ca­tion­s. And yet, even to­day, we write pro­gram af­ter pro­gram that is closed--that its users can­not pro­gram--that ad­mits of no ex­ten­sions with­out our adding to its source code. If we be­lieve in­tel­lec­tu­al­ly, from a sound un­der­stand­ing of the eco­nom­ic in­cen­tives in the mar­ket­place, that open sys­tems are bet­ter, and have seen this be­lief con­firmed re­peat­ed­ly in the mar­ket­place, then the on­ly ques­tion that re­mains is why? Why not make ev­ery pro­gram an open pro­gram?

And this is in­deed very im­por­tan­t. Any app worth de­vel­op­ing is prob­a­bly worth de­vel­op­ing in an ex­ten­si­ble man­ner. Af­ter al­l, you are prob­a­bly nev­er go­ing to fig­ure out what the us­er re­al­ly wants to do.

But why is not ev­ery pro­gram "open" in this man­ner?

Well, be­cause it's HARD! Writ­ing a closed pro­gram has tra­di­tion­al­ly been much less work at ev­ery stage of the de­vel­op­ment cy­cle: eas­i­er to de­sign, less code to write, sim­pler doc­u­men­ta­tion, and far few­er con­sid­er­a­tions in the test phase. In ad­di­tion, closed prod­ucts are be­lieved to be less de­mand­ing of sup­port, al­though I'll ar­gue lat­er that this as­sump­tion may be in­cor­rec­t.

This is true, al­though it was much more true in 1991 when At­last was re­leased to the pub­lic do­main. Nowa­days the gen­er­al­ized adop­tion of more dy­nam­ic lan­guages has made this much eas­i­er. Af­ter al­l, you can write a plug­in sys­tem for your python app in per­haps ten lines of code.

How­ev­er, there is a strong back­lash from the pains of MSOf­fice script­ing, which in­volved love­li­ness like func­tion names that changed ac­cord­ing to your lo­cale, and ba­sic as the lan­guage of choice.

On stat­ic lan­guages this is hard­er, but there are tech­no­log­i­cal so­lu­tion­s. For ex­am­ple, in KDE there is al­ready a lim­it­ed script­abil­i­ty cul­ture based on how easy it is to pro­vide ex­ter­nal DCOP in­ter­faces (I sup­pose I should say DBUS nowa­days). Al­so for KDE4 the Kross script­ing frame­work prom­ises lan­guage-neu­tral script­ing.

Most pro­grams start out as non­pro­grammable, closed ap­pli­ca­tion­s, then painful­ly claw their way to pro­gramma­bil­i­ty through the in­tro­duc­tion of a lim­it­ed script or macro fa­cil­i­ty, suc­ceed­ed by an in­creas­ing­ly com­pre­hen­sive in­ter­pre­tive macro lan­guage which grows like top­sy and with­out a co­her­ent de­sign as us­er de­mands up­on it grow. Fi­nal­ly, per­hap­s, the pro­gram is out­fit­ted with bind­ings to ex­ist­ing lan­guages such as C.

Noone pro­vides straight cus­tom C bind­ings for ap­pli­ca­tions now, right? But yes, I ex­pect script­ing is not the first thing that gets im­ple­ment­ed.

An al­ter­na­tive to this is adopt­ing a stan­dard lan­guage as the macro lan­guage for a prod­uc­t. This ap­proach has many at­trac­tion­s. First, choos­ing a stan­dard lan­guage al­lows users to avail them­selves of ex­ist­ing books and train­ing re­sources to learn its ba­sic­s. The de­vel­op­er of a ded­i­cat­ed macro lan­guage must cre­ate all this ma­te­ri­al from scratch. Sec­ond, an in­ter­pre­tive lan­guage, where all pro­grams are rep­re­sent­ed in ASCII code, is in­her­ent­ly por­ta­ble across com­put­ers and op­er­at­ing sys­tem­s. Once the in­ter­preter is got­ten to work on a new sys­tem, all the pro­grams it sup­ports are pret­ty much guar­an­teed to work. Third, most ex­ist­ing lan­guages have evolved to the point that most of the rough edges have been tak­en off their de­sign. Ex­tend­ing an ex­ist­ing lan­guage along the lines laid down by its de­sign­ers is much less like­ly to re­sult in an in­com­pre­hen­si­ble dis­as­ter than grow­ing an ad-hoc macro lan­guage fea­ture by neat-o fea­ture.

Un­for­tu­nate­ly, in­ter­preters are slow, slow, slow. A sim­ple cal­cu­la­tion of the num­ber of in­struc­tions of over­head per in­struc­tion that fur­thers the ex­e­cu­tion of the pro­gram quick­ly demon­strates that no in­ter­preter is suit­able for se­ri­ous com­pu­ta­tion. As long as the in­ter­preter is de­ployed in the role of a macro lan­guage, this may not be a sub­stan­tial con­sid­er­a­tion. How­ev­er, as soon as ap­pli­ca­tions try to do sub­stan­tial com­pu­ta­tion, the over­head of an in­ter­preter be­comes a crush­ing bur­den, verg­ing on in­tol­er­a­ble. The ob­vi­ous al­ter­na­tive is to pro­vide a com­piled lan­guage. But that, too, has its prob­lem­s.

In­ter­est­ing point here is that again, the speed of in­ter­preters is not so big a deal nowa­days. Af­ter al­l, Lu­a­JIT is pret­ty fast for most us­es.

Us­ing a com­pil­er as an ex­ten­sion lan­guage is not some­thing I have run in­to but I could do some pret­ty things us­ing, for ex­am­ple, TCC, Python and python-in­stant.

Now we get to AT­LAST it­self:

AT­LAST is a tool­kit that makes ap­pli­ca­tions pro­gram­mable. De­lib­er­ate­ly de­signed to be easy to in­te­grate both in­to ex­ist­ing pro­grams and new­ly-de­vel­oped ones, AT­LAST pro­vides any pro­gram that in­cor­po­rates it most of the ben­e­fits of pro­gramma­bil­i­ty with very lit­tle ex­plic­it ef­fort on the part of the de­vel­op­er. In­deed, once you be­gin to "think AT­LAST" as part of the de­sign cy­cle, you'll prob­a­bly find that the way you de­sign and build pro­grams changes sub­stan­tial­ly. I'm com­ing to think of AT­LAST as the "mon­ster that feeds on pro­gram­s," be­cause in­clud­ing it in a pro­gram tends to shrink the amount of spe­cial-pur­pose code that would oth­er­wise have to be writ­ten while re­sult­ing in fin­ished ap­pli­ca­tions that are open, ex­ten­si­ble, and more eas­i­ly adapt­ed to oth­er op­er­at­ing en­vi­ron­ments such as the event driv­en par­a­digm.

The idea of a por­ta­ble toolk­it, in­te­grat­ed in­to a wide va­ri­ety of prod­uct­s, all of which there­by share a com­mon pro­gram­ming lan­guage seems ob­vi­ous once you con­sid­er its ad­van­tages. It's sur­pris­ing that such pack­ages aren't com­mon­place in the in­dus­try. In fac­t, the on­ly true an­tecedent to AT­LAST I've en­coun­tered in my whole twist­ed path through this in­dus­try was the uni­ver­sal macro pack­age de­vel­oped in the mid 1970's by Kern Sib­bald and Ben Cranston at the Uni­ver­si­ty of Mary­land. That pack­age, im­ple­ment­ed on Uni­vac main­frames, pro­vid­ed a com­mon macro lan­guage shared by a wide va­ri­ety of Uni­ver­si­ty of Mary­land util­i­ties, in­clud­ing a text ed­i­tor, de­bug­ger, file dumper, and type­set­ting lan­guage. While AT­LAST is en­tire­ly dif­fer­ent in struc­ture and op­er­a­tion from the Mary­land pack­age, which was an in­ter­pre­tive string lan­guage, the con­cept of a cross-prod­uct macro lan­guage and ap­pre­ci­a­tion of the ben­e­fits to be had from such a pack­age are di­rect­ly trace­able to those root­s.

This con­cept was lat­er adopt­ed by Lu­a, Tcl, and you could use Python in this man­ner, al­though usu­al­ly it's done the oth­er way around. Which means that there is not so much shar­ing of ex­ten­sion lan­guages as there could be.

And on­to the con­clu­sion­s:

Ev­ery­thing should be pro­gram­mable. EV­ERY­THING! I have come to the con­clu­sion that to write al­most any pro­gram in a closed man­ner is a mis­take that in­vites the ex­pen­di­ture of un­count­ed hours "en­hanc­ing" it over its life cy­cle. Fur­ther tweak­s, "fea­tures," and "fix­es" of­ten re­sult in a prod­uct so mas­sive and in­com­pre­hen­si­ble that it be­comes un­learn­able, un­main­tain­able, and even­tu­al­ly un­us­able.

Amen.

Far bet­ter to in­vest the ef­fort up front to cre­ate a prod­uct flex­i­ble enough to be adapt­ed at will, by its user­s, to their im­me­di­ate need­s. If the prod­uct is pro­gram­mable in a portable, open for­m, us­er ex­ten­sions can be ex­changed, com­pared, re­viewed by the prod­uct de­vel­op­er, and even­tu­al­ly in­cor­po­rat­ed in­to the main­stream of the prod­uc­t.

This pre­fig­ures the cur­rent FLOSS ecosys­tem, which is pret­ty im­pres­sive for a prod­uct that was ma­ture in 1991, al­though of course there was al­ready a com­mu­ni­ty of EMACS hack­ing and sim­i­lar nich­es.

It is far, far bet­ter to have thou­sands of cre­ative users ex­pand­ing the scope of one's prod­uct in ways the orig­i­nal de­vel­op­ers did­n't an­tic­i­pate--in fac­t, work­ing for the ven­dor with­out pay, than it is to have thou­sands of frus­trat­ed users writ­ing up wish list re­quests that the ven­dor can com­ply with on­ly by hir­ing peo­ple and pay­ing them to try to ac­com­mo­date the per­ceived needs of the user­s.

True if a lit­tle too cyn­i­cal for my cur­rent taste ;-) Of course in FLOSS there is no such con­flict be­tween users and de­vel­op­ers be­cause the de­vel­op­ers can usu­al­ly just whine ex­­plain to the users the re­al­i­ties of free soft­­ware de­vel­op­­men­t. But hey, a low bar­ri­er to en­try is al­ways a nice thing to have.

Open ar­chi­tec­ture and pro­gramma­bil­i­ty not on­ly ben­e­fits the user, not on­ly makes a prod­uct bet­ter in the tech­ni­cal and mar­ket­ing sense, but con­fers a di­rect eco­nom­ic ad­van­tage up­on the ven­dor of such a pro­duc­t--one mir­rored in a com­men­su­rate dis­ad­van­tage to the ven­dor of a closed prod­uc­t.

The chief ar­gu­ment against pro­gramma­bil­i­ty has been the ex­tra in­vest­ment need­ed to cre­ate open prod­uct­s. AT­LAST pro­vides a way of build­ing open prod­ucts in the same, or less, time than it takes to con­struct closed ones. Just as no C pro­gram­mer in his right mind would sit down and write his own buffered file I/O pack­age when a per­fect­ly fine one was sit­ting in the li­brary, why re-in­vent a macro lan­guage or oth­er pa­ram­e­ter­i­sa­tion and pro­gram­ming fa­cil­i­ty when there's one just sit­ting there that's as fast as na­tive C code for all but the most ab­surd mis­ap­pli­ca­tion­s, takes less than 51K with ev­ery gew-­gaw and op­tion­al fea­ture at its com­mand en­abled all at on­ce, is por­ta­ble to any ma­chine that sup­ports C by sim­ply re­com­pil­ing a sin­gle file, and can be in­te­grat­ed in­to a typ­i­cal ap­pli­ca­tion at a ba­sic lev­el in less than 15 min­utes?

And then pro­ceeds to throw a Forth vari­ant at you ;-) Good con­cep­t, per­haps not the nicest lan­guage to use, al­though the choice is very un­der­stand­able.

Am I propos­ing that ev­ery ap­pli­ca­tion sud­den­ly look like FORTH? Of course not; no more than out­put from Post­Script print­ers looks like PostScrip­t, or ap­pli­ca­tions that run on 80386 pro­ces­sors re­sem­ble 80386 as­sem­bly lan­guage. AT­LAST is an in­ter­me­di­ate lan­guage, seen on­ly by those en­gaged in im­ple­ment­ing and ex­tend­ing the prod­uc­t. Even then, AT­LAST is a chameleon which, with prop­er­ly de­fined word­s, can look like al­most any­thing you like, even at the prim­i­tive lev­el of the in­ter­preter.

Again and again, I have been faced with de­sign sit­u­a­tions where I knew that I re­al­ly need­ed pro­gramma­bil­i­ty, but did­n't have the time, the mem­o­ry, or the for­ti­tude to face the prob­lem square­ly and solve it the right way. In­stead, I end­ed up cre­at­ing a kludge that con­tin­ued to bur­den me through time. This is just a high­er lev­el man­i­fes­ta­tion of the night­mares per­pe­trat­ed by old-­time pro­gram­mers who did­n't have ac­cess to a prop­er dy­nam­ic mem­o­ry al­lo­ca­tor or linked list pack­age. Just be­cause pro­gramma­bil­i­ty is the mag­ic smoke of com­put­ing does­n't mean we should be spooked by the ghost in the ma­chine or hes­i­tant to con­fer its pow­er up­on our cus­tomer­s.

Oh yes. Li­braries rule, and when there is no such thing as the li­brary you need, it's the worst po­si­tion to be in.

Don't think of AT­LAST as FORTH. Don't think of it as a lan­guage at al­l. The best way to think of AT­LAST is as a li­brary rou­tine that gives you pro­gramma­bil­i­ty, in the same sense oth­er li­braries pro­vide file ac­cess, win­dow man­age­men­t, or graph­ics fa­cil­i­ties. The whole con­cept of "pro­gramma­bil­i­ty in a can" is odd­--it took me two years from the time I first thought about it un­til I re­al­ly got my end ef­fec­tor around it and crushed it in­to sub­mis­sion.

I am prob­a­bly go­ing to use AT­LAST or some­thing sim­i­lar in a pro­gram, but the us­er will not see a forth-­like lan­guage at al­l, but a clas­si­cal Ex­cel-­like for­mu­la lan­guage which would get com­piled to the for­thish lan­guage be­hind the scenes.

Open is bet­ter. AT­LAST lets you build open pro­grams in less time than you used to spend writ­ing closed ones. Pro­grams that in­her­it their open ar­chi­tec­ture from AT­LAST will share, across the en­tire prod­uct line and among all hard­ware plat­forms that sup­port it, a com­mon, clean, and ef­fi­cient means of us­er ex­ten­si­bil­i­ty. The po­ten­tial ben­e­fits of this are im­mense.

In­deed. Prob­a­bly not us­ing AT­LAST, but Lua or Python or some­thing else in­stead. And stil­l, 16 years af­ter AT­LAST was re­leased to the pub­lic do­main, we are still walk­ing down this road, and not even close to the goal.