I have been exploring embeddable languages for the last month or so. I have learned forth and some of its many many many variants and while exploring one of the most obscure ones called Atlast , I found a very interesting README which I will quote liberally in this post.
Virtually every industry analyst agrees that open architecture is
essential to the success of applications. And yet, even today, we
write program after program that is closed--that its users cannot
program--that admits of no extensions without our adding to its source
code. If we believe intellectually, from a sound understanding of the
economic incentives in the marketplace, that open systems are better,
and have seen this belief confirmed repeatedly in the marketplace,
then the only question that remains is why? Why not make every
program an open program?
And this is indeed very important. Any app worth developing is probably worth
developing in an extensible manner. After all, you are probably never going to
figure out what the user really wants to do.
But why is not every program "open" in this manner?
Well, because it's HARD! Writing a closed program has traditionally
been much less work at every stage of the development cycle: easier to
design, less code to write, simpler documentation, and far fewer
considerations in the test phase. In addition, closed products are
believed to be less demanding of support, although I'll argue later
that this assumption may be incorrect.
This is true, although it was much more true in 1991 when Atlast was released to
the public domain. Nowadays the generalized adoption of more dynamic languages
has made this much easier. After all, you can write a plugin system for your python app
in perhaps ten lines of code.
However, there is a strong backlash from the pains of MSOffice scripting, which involved
loveliness like function names that changed according to your locale, and basic as
the language of choice.
On static languages this is harder, but there are technological solutions. For example, in
KDE there is already a limited scriptability culture based on how easy it is to provide
external DCOP interfaces (I suppose I should say DBUS nowadays). Also for KDE4 the Kross
scripting framework promises language-neutral scripting.
Most programs start out as nonprogrammable, closed applications, then
painfully claw their way to programmability through the introduction
of a limited script or macro facility, succeeded by an increasingly
comprehensive interpretive macro language which grows like topsy and
without a coherent design as user demands upon it grow. Finally,
perhaps, the program is outfitted with bindings to existing languages
such as C.
Noone provides straight custom C bindings for applications now, right? But yes,
I expect scripting is not the first thing that gets implemented.
An alternative to this is adopting a standard language as the macro
language for a product. This approach has many attractions. First,
choosing a standard language allows users to avail themselves of
existing books and training resources to learn its basics. The
developer of a dedicated macro language must create all this material
from scratch. Second, an interpretive language, where all programs
are represented in ASCII code, is inherently portable across computers
and operating systems. Once the interpreter is gotten to work on a
new system, all the programs it supports are pretty much guaranteed to
work. Third, most existing languages have evolved to the point that
most of the rough edges have been taken off their design. Extending
an existing language along the lines laid down by its designers is
much less likely to result in an incomprehensible disaster than
growing an ad-hoc macro language feature by neat-o feature.
Unfortunately, interpreters are slow, slow, slow. A simple
calculation of the number of instructions of overhead per instruction
that furthers the execution of the program quickly demonstrates that
no interpreter is suitable for serious computation. As long as the
interpreter is deployed in the role of a macro language, this may not
be a substantial consideration. However, as soon as applications try
to do substantial computation, the overhead of an interpreter becomes
a crushing burden, verging on intolerable. The obvious alternative is
to provide a compiled language. But that, too, has its problems.
Interesting point here is that again, the speed of interpreters is not so
big a deal nowadays. After all, LuaJIT is pretty fast for most uses.
Using a compiler as an extension language is not something I have run into
but I could do some pretty things using, for example, TCC, Python and python-instant.
Now we get to ATLAST itself:
ATLAST is a toolkit that makes applications programmable.
Deliberately designed to be easy to integrate both into existing
programs and newly-developed ones, ATLAST provides any program that
incorporates it most of the benefits of programmability with very
little explicit effort on the part of the developer. Indeed, once you
begin to "think ATLAST" as part of the design cycle, you'll probably
find that the way you design and build programs changes substantially.
I'm coming to think of ATLAST as the "monster that feeds on programs,"
because including it in a program tends to shrink the amount of
special-purpose code that would otherwise have to be written while
resulting in finished applications that are open, extensible, and more
easily adapted to other operating environments such as the event
driven paradigm.
The idea of a portable toolkit, integrated into a wide variety of
products, all of which thereby share a common programming language
seems obvious once you consider its advantages. It's surprising that
such packages aren't commonplace in the industry. In fact, the only
true antecedent to ATLAST I've encountered in my whole twisted path
through this industry was the universal macro package developed in the
mid 1970's by Kern Sibbald and Ben Cranston at the University of
Maryland. That package, implemented on Univac mainframes, provided a
common macro language shared by a wide variety of University of
Maryland utilities, including a text editor, debugger, file dumper,
and typesetting language. While ATLAST is entirely different in
structure and operation from the Maryland package, which was an
interpretive string language, the concept of a cross-product macro
language and appreciation of the benefits to be had from such a
package are directly traceable to those roots.
This concept was later adopted by Lua, Tcl, and you could use Python
in this manner, although usually it's done the other way around. Which means
that there is not so much sharing of extension languages as there could be.
And onto the conclusions:
Everything should be programmable. EVERYTHING! I have come to the
conclusion that to write almost any program in a closed manner is a
mistake that invites the expenditure of uncounted hours "enhancing" it
over its life cycle. Further tweaks, "features," and "fixes" often
result in a product so massive and incomprehensible that it becomes
unlearnable, unmaintainable, and eventually unusable.
Amen.
Far better to invest the effort up front to create a product flexible
enough to be adapted at will, by its users, to their immediate needs.
If the product is programmable in a portable, open form, user
extensions can be exchanged, compared, reviewed by the product
developer, and eventually incorporated into the mainstream of the
product.
This prefigures the current FLOSS ecosystem, which is pretty impressive for
a product that was mature in 1991, although of course there was already
a community of EMACS hacking and similar niches.
It is far, far better to have thousands of creative users expanding
the scope of one's product in ways the original developers didn't
anticipate--in fact, working for the vendor without pay, than it is to
have thousands of frustrated users writing up wish list requests that
the vendor can comply with only by hiring people and paying them to
try to accommodate the perceived needs of the users.
True if a little too cynical for my current taste ;-) Of course in FLOSS
there is no such conflict between users and developers because the
developers can usually just whine explain to
the users the realities of free software development. But hey, a low barrier
to entry is always a nice thing to have.
Open
architecture and programmability not only benefits the user, not only
makes a product better in the technical and marketing sense, but
confers a direct economic advantage upon the vendor of such a
product--one mirrored in a commensurate disadvantage to the vendor of
a closed product.
The chief argument against programmability has been the extra
investment needed to create open products. ATLAST provides a way of
building open products in the same, or less, time than it takes to
construct closed ones. Just as no C programmer in his right mind
would sit down and write his own buffered file I/O package when a
perfectly fine one was sitting in the library, why re-invent a macro
language or other parameterisation and programming facility when
there's one just sitting there that's as fast as native C code for all
but the most absurd misapplications, takes less than 51K with every
gew-gaw and optional feature at its command enabled all at once, is
portable to any machine that supports C by simply recompiling a single
file, and can be integrated into a typical application at a basic
level in less than 15 minutes?
And then proceeds to throw a Forth variant at you ;-) Good concept, perhaps
not the nicest language to use, although the choice is very understandable.
Am I proposing that every application suddenly look like FORTH? Of
course not; no more than output from PostScript printers looks like
PostScript, or applications that run on 80386 processors resemble
80386 assembly language. ATLAST is an intermediate language, seen
only by those engaged in implementing and extending the product. Even
then, ATLAST is a chameleon which, with properly defined words, can
look like almost anything you like, even at the primitive level of the
interpreter.
Again and again, I have been faced with design situations where I knew
that I really needed programmability, but didn't have the time, the
memory, or the fortitude to face the problem squarely and solve it the
right way. Instead, I ended up creating a kludge that continued to
burden me through time. This is just a higher level manifestation of
the nightmares perpetrated by old-time programmers who didn't have
access to a proper dynamic memory allocator or linked list package.
Just because programmability is the magic smoke of computing doesn't
mean we should be spooked by the ghost in the machine or hesitant to
confer its power upon our customers.
Oh yes. Libraries rule, and when there is no such thing as the library you
need, it's the worst position to be in.
Don't think of ATLAST as FORTH. Don't think of it as a language at
all. The best way to think of ATLAST is as a library routine that
gives you programmability, in the same sense other libraries provide
file access, window management, or graphics facilities. The whole
concept of "programmability in a can" is odd--it took me two years
from the time I first thought about it until I really got my end
effector around it and crushed it into submission.
I am probably going to use ATLAST or something similar in a program, but
the user will not see a forth-like language at all, but a classical
Excel-like formula language which would get compiled to the forthish
language behind the scenes.
Open is better. ATLAST lets you build open programs in less time than
you used to spend writing closed ones. Programs that inherit their
open architecture from ATLAST will share, across the entire product
line and among all hardware platforms that support it, a common,
clean, and efficient means of user extensibility. The potential
benefits of this are immense.
Indeed. Probably not using ATLAST, but Lua or Python or something else instead.
And still, 16 years after ATLAST was released to the public domain, we are still
walking down this road, and not even close to the goal.