Two up­com­ing fea­tures for the next re­lease of Niko­la, my stat­ic site gen­er­a­tor, due some­time in Au­gust.

FILTERS = {
    ".css": [filters.yui_compressor],
    ".js": [filters.yui_compressor],
}

FILTERS = {
    ".jpg": ["jpegoptim '%s'"],
    ".png": ["pngoptim '%s'"],
}

I had pro­posed a talk for Py­Con Ar­genti­na called "Driv­ing 3 Nails with a Shoe". I know, the ti­tle is sil­ly, but the idea was show­ing how to do things us­ing the wrong tool, in­ten­tion­al­ly. Why? Be­cause:

1. It makes you think dif­fer­­ent

2. It's fun

The bad side is, of course, that this talk's con­tents have to be a se­cret, or else the fun is spoiled for ev­ery­one. Since the re­view process for Py­ConAr talks is pub­lic, there was no way to ex­plain what this was about.

And since that means the re­view­ers ba­si­cal­ly have to take my word for this be­ing a good thing to have at a con­fer­ence, which is un­fair, I delet­ed the pro­pos­al. The good (may­be) news is that now ev­ery­one will see what those ideas I had were about. And here is nail num­ber 1: Writ­ing a spread­sheet us­ing doit.

This is not my first "spread­sheet". It all start­ed a long, long time ago with a fa­mous recipe by Ray­mond Het­tinger which I used again and again and again (I may even be miss­ing some post there).

Since I have been us­ing doit for Niko­la I am im­pressed by the pow­er it gives you. In short, doit lets you cre­ate tasks, and those tasks can de­pend on oth­er tasks, and op­er­ate on data, and pro­vide re­sults for oth­er tasks, etc.

See where this is go­ing?

So, here's the code, with ex­pla­na­tion­s:

cells is our spreadsheet. You can put anything there, just always use "cellname=formula" format, and the formula must be valid Python, ok?

from tokenize import generate_tokens

cells = ["A1=A3+A2", "A2=2", "A3=4"]
values = {}

task_calculate creates a task for each cell, called calculate:CELLNAME. The "action" to be performed by that task is evaluating the formula. But in order to do that successfully, we need to know what other cells have to be evaluated first!

This is im­ple­ment­ed us­ing doit's cal­cu­lat­ed de­pen­den­cies by ask­ing doit to run the task "get_de­p:­FOR­MU­LA" for this cel­l's for­mu­la.

def evaluate(name, formula):
    value = eval(formula, values)
    values[name] = value
    print "%s = %s" % (name, value)

def task_calculate():
    for cell in cells:
        name, formula = cell.split('=')
        yield {
            'name':name,
            'calc_dep': ['get_dep:%s' % formula],
            'actions': [(evaluate, (name, formula))],
            }

For example, in our test sheet, A1 depends on A3 and A2 but those depend on no other cells. To figure this out, I will use the tokenize module, and just remember what things are "names". More sophisticated approaches exist.

The task_get_dep function is a doit task that will create a task called "get_dep:CELLNAME" for every cell name in cells.

What get_dep returns is a list of doit tasks. For our A1 cell, that would be ["calculate:A2", "calculate:A3"] meaning that to calculate A1 you need to perform those tasks first.

def get_dep(formula):
    """Given a formula, return the names of the cells referenced."""
    deps = {}
    try:
        for token in generate_tokens([formula].pop):
            if token[0] == 1:  # A variable
                deps[token[1]] = None
    except IndexError:
        # It's ok
        pass
    return {
        'result_dep': ['calculate:%s' % key for key in deps.keys()]
        }

def task_get_dep():
    for cell in cells:
        name, formula = cell.split('=')
        yield {
            'name': formula,
            'actions': [(get_dep, (formula,))],
            }

And that's it. Let's see it in action. You can get your own copy here and try it out by installing doit, editing cells and then running it like this:

ralsina@perdido:~/dosheet$ doit -v2 calculate:A3
.  get_dep:4
{}
.  calculate:A3
A3 = 4
ralsina@perdido:~/dosheet$ doit -v2 calculate:A2
.  get_dep:2
{}
.  calculate:A2
A2 = 2
ralsina@perdido:~/dosheet$ doit -v2 calculate:A1
.  get_dep:A3+A2
{'A3': None, 'A2': None}
.  get_dep:4
{}
.  calculate:A3
A3 = 4
.  get_dep:2
{}
.  calculate:A2
A2 = 2
.  calculate:A1
A1 = 6

As you can see, it al­ways does the min­i­mum amount of ef­fort to cal­cu­late the de­sired re­sult. If you are so in­clined, there are some things that could be im­proved, and I am leav­ing as ex­er­cise for the read­er, for ex­am­ple:

1. Use up­­­to­­date to avoid re­­cal­cu­lat­ing de­pen­­den­­cies.

2. Get rid of the glob­al val­ues and use doit's com­put­ed val­ues in­stead.

Here is the full list­ing, en­joy!

This may cause some pal­pi­ta­tions in some friends of mine who laugh at me for us­ing kwrite, but it re­al­ly is not. Any time you spend con­fig­ur­ing, choos­ing, ad­just­ing, tweak­ing, chang­ing, im­prov­ing, patch­ing or get­ting used to your ed­i­tor is time in­vest­ed, for which you need to show a ben­e­fit, or else it's time wast­ed.

Let's look at SLOC, which while dis­cred­it­ed as a mea­sure of pro­gram­mer's pro­duc­tiv­i­ty, sure­ly does work as a mea­sure of how much a pro­gram­mer types, right?

Well, es­ti­mates of to­tal code pro­duc­tion across the life­time of a prod­uct vary (just take the SLOC of the pro­duc­t, di­vide by men/­days spen­t), but they are usu­al­ly some­thing be­tween 10 and 100 SLOC per pro­gram­mer per day. Let's be gen­er­ous and say 200.

So, 200 lines in eight hours. That's rough­ly one line ev­ery two min­utes, and the av­er­age line of code is about 45 char­ac­ter­s. Since I as­sume you are a com­pe­tent typ­ist (if you are not, shame on you!), it takes less than 20 sec­onds to type that.

So, typ­ing, which is what you of­ten tweak in your ed­i­tor, takes less than 15% of your time. And how much faster can it get? Can you get a line writ­ten in 10 sec­ond­s? Then you just saved 8% of your day. And re­al­ly, does your ed­i­tor isave you half the typ­ing time?

How much time do you lose hav­ing your eyes won­der over the side­bars, but­ton­s, tool­bars, etc?

So while yes, typ­ing faster and more ef­fi­cient­ly is an op­ti­miza­tion, it may al­so be pre­ma­ture, in that, what the hell are we do­ing the oth­er 80% of the time? Is­n't there some­thing we can do to make that huge chunck of time more ef­fi­cient in­stead of the small­er chunk?

Well, I think we spent most of that time do­ing three things:

1. Read­­ing code

2. Think­ing about what code to write

3. Fix­ing what we wrote in that oth­­er 20%

The first is easy: we need bet­ter code read­ers not ed­i­tors. It's a pity that the main in­ter­face we get for look­ing at code is an ed­i­tor, with its con­stant lure to­wards just chang­ing stuff. I think there is a lost op­por­tu­ni­ty there some­where, for an app where you can look at the code in orig­i­nal or in­ter­est­ing ways, so that you un­der­stand the code bet­ter.

The sec­ond is hard­er, be­cause it's per­son­al. I walk. If you see me walk­ing while my ed­i­tor is open, I am think­ing. Af­ter I think, I write. Your mileage may vary.

The third is by far the hard­est of the three. For ex­am­ple, au­to­com­plete helps there, be­cause you won't mistype things, which is in­ter­est­ing, but more pow­er­ful ap­proach­es ex­ist, like con­stant run­ning of tests suites while you ed­it. Ev­ery time you leave a line, trig­ger the af­fect­ed parts of the suit­e.

That's much hard­er than it sound­s, since it means your tools need to cor­re­late your test suite to your code very tight­ly, so that you will see you are break­ing stuff the sec­ond you break it, not min­utes lat­er.

Al­so, it should en­able you to jump to the test with a keystroke, so that you can fix those tests if you are chang­ing be­hav­iour in your code. And of course it will mean you need tests ;-)

Which brings me to a pet peeve of mine, that ed­i­tors still treat the file as the unit of work, which makes no sense at al­l. You nev­er want to ed­it a file, you want to ed­it a func­tion, or a class, or a method, or a con­stant but nev­er a file. Know­ing this was the sheer ge­nius of an­cient Vis­ual Ba­sic, which was com­plete­ly ig­nored by all the snobs look­ing down at it.

So, in­stead of tweak­ing your ed­i­tor, get me a tool that does what I need please. I have been wait­ing for it since VB 1.0. And a sand­wich.

UP­DATE: In­ter­est­ing dis­cus­sions in red­dit and hack­er news

If you have ac­tu­al, pa­per books and want to cat­a­log their in­fo quick­ly, this book­da­ta.py script may be handy:

import sys
import time
import gdata.books.service
import json

def get_book_info(isbn):
    print "Looking for ISBN:", isbn
    google_books = gdata.books.service.BookService()
    result = google_books.search('ISBN %s '%isbn)
    data = [x.to_dict() for x in result.entry]
    if not data:
        print "No results"
        return
    title = data[0]['title']
    with open(title+'.json','w') as f:
        f.write(json.dumps(data))
    print "Guardada info de '%s' en '%s.json'" %(isbn, title)

if __name__ == "__main__":
    while True:
        isbn = sys.stdin.readline().strip()
        if isbn:
            get_book_info(isbn)
        time.sleep(1)

What does it do? It reads IS­BN num­bers from stan­dard in­put and saves the book's in­fo in a ti­tle.j­son file for lat­er pro­cess­ing and for­mat­ting.

If you want to ed­it that in­for­ma­tion, you can just do it or you can try do­ing a lit­tle script us­ing json­wid­get like this:

python -c 'import jsonwidget; jsonwidget.run_editor("abook.json", schemafile="gbooks.schema")'

Where abook.j­son is a file gen­er­at­ed by the pre­vi­ous script and gbook­s.schema is this file.

Oh, and if your books have bar­codes, you can just do:

zbarcam --raw | python bookdata.py

Show your com­put­er your books and let it do the rest :-)

PS: I would love if some­one gath­ered all this and made a nice per­son­al book cat­a­logu­ing thing.

As I have men­tioned in half a dozen posts al­ready, I spent the last week­end at Py­Camp 2012. But what I have not writ­ten about is what ex­act­ly it was, and why any­one would want to at­tend one, or maybe or­ga­nize one. So that's what this post is about.

Py­Camp was or­ga­nized by PyAr, the Python Ar­genti­na com­mu­ni­ty. PyAr is a very spe­cial bunch of peo­ple, who are com­plete­ly am­a­teur, and do ev­ery­thing for love and fun. Since PyAr is a very spe­cial group of peo­ple, the things PyAr caus­es, in­spires or cre­ates are spe­cial as well.

So, since a few years ago, what hap­pens is some­one finds a place with bunk bed­s, a large room, per­haps some­what iso­lat­ed, that pro­vides meal­s, and is cheap (it's as hard as it sound­s) and rents it for a long week­end. Then ev­ery­one is in­vit­ed to chip in for the rent mon­ey.

This year, 4-­days, all in­clu­sive, cost­ed rough­ly U$S 100. Sure, it's not ex­act­ly lux­u­ry ac­co­mo­da­tion­s, but it does what it has to do, which is give us shel­ter and pro­tect us from wild an­i­mal­s.

Thus, you end up with a few dozen nerds with com­put­er­s, one of them is great at set­ting up wire­less (Joac!), one is the MC (Ale­cu!), one helps around (Fa­cun­do!) one is the li­ai­son with the lo­ca­tion (Pin­don­ga!) and so on, the work is spread around, and we have time and com­pa­ny to hack.

So, on the first morn­ing, ev­ery­one pro­pos­es what he would like to work on. Those pro­pos­als are vot­ed by the pub­lic, and those with more votes are as­signed slots (5 a day), where they will be the main fo­cus of at­ten­tion.

So, what hap­pens if your pro­pos­al is not vot­ed? Well, you ei­ther find a pro­pos­al you like, and join it, or you just do your thing. Be­cause this is not a democ­ra­cy, this is an­ar­chy, the votes are just a way for ev­ery­one to know what peo­ple will be do­ing, and to find places to fit in if you want (BTW, there is a sit­u­a­tion in LeGuin's The Dis­posessed which is so much like this, it's scary).

Af­ter that, you just do what you wan­t. You can put your head­set on, and code, or min­gle and chat, or join a group, or do a bit of ev­ery­thing. Since meals are catered, you don't have to wor­ry about break­s. When the meal is ready, ev­ery­one breaks at the same time and so­cial­izes in com­mu­ni­tary ta­bles.

Does all this sound as strange to you as it does to me? A bunch of grown pro­fes­sion­als act­ing like hip­pies. Well, it feels strange too, but that does­n't mean it does­n't feel great. It even works great. Once you see what the oth­ers are do­ing, things you would­n't ex­pect start look­ing like fun (Cel­ery!?! Jug­ger­naut! An­droid!) and the sheer ex­cite­ment of peo­ple telling you "look, I did this!" is in­fec­tious, and ex­hil­a­rat­ing.

Al­so, RC cars, kinect hack­ing, an­droid hack­ing, elec­tric gui­tar hack­ing, jug­gling, mono­cy­cle lesson­s, a firepit, al­co­hol, cof­fee, mate, boardgames, cardgames, mu­sic, jokes, adrenaline, huge spi­der­s, asado, cold, vim, nin­ja, ping pong, ro­bot space­ship­s, peo­ple you see on­ly twice a year if that, ques­tion­s, not know­ing the an­swer­s, fig­ur­ing things out on the run, get­ting help in that thing you have been stuck for week­s, hav­ing the piece some­one else has been stuck on for week­s, feel­ing like some sort of beard­ed bud­dha and a to­tal ig­no­ra­mus in 5 min­utes...

And at least I, at least this year, had a very pro­duc­tive week­end. I got help from a bunch of peo­ple in things I was daunt­ed by, I felt like an ac­tive pro­gram­mer in­stead of a suit, which is al­ways nice, since I don't own a suit, and had a great time. Laughed a lot. Made a cou­ple new friend­s. Saw a bunch of old ones. Helped a few peo­ple.

So, I would like oth­er peo­ple to have as great a time as I had. Of course com­ing to Ar­genti­na is prob­a­bly not a great idea. It's an ex­pen­sive trip, if you don't speak span­ish you will miss a lot, and if Py­Camp gets too big it may stop be­ing fun at al­l.

But why not do some­thing sim­i­lar? Does­n't have to be about Python, you can do it about mak­ing stuff, about pro­gram­ming in gen­er­al, what­ev­er. Just get a some­what com­fort­able, some­what iso­lat­ed place with a rea­son­able cater­ing and get your 50 near­est geeks there, and have a ton of fun.

You may get some­thing use­ful done, too.