This fri­day is the be­gin­ning of Py­Cam­p, four days of python hack­ing with­out dis­trac­tion or pause. And I want to code a lot. My main tar­get is fea­tures for Niko­la my stat­ic blog gen­er­a­tor.

If you are at­tend­ing Py­Camp (or even if you are not), you are wel­come to join me in im­ple­ment­ing these in a marathon of kick­ass cod­ing start­ing this fri­day and last­ing all week­end.

I have a few ideas in my head, but I want more. These are the ones I have, please add more in the com­ments if you have any:

Code Gallery

Like im­age gal­leries but for code. Put code in a fold­er and it will be beau­ti­ful­ly dis­played. With the ad­di­tion of a "list­ings" do­cu­tils di­rec­tive, it will make show­ing code in de­tail and in con­text easy and pow­er­ful, and make Niko­la more at­trac­tive to pro­gram­mer-blog­ger­s.

Gallery Polishing

Im­age gal­leries are im­ple­ment­ed and work, but they could use a lot of pol­ish. From mak­ing them more net­work-­ef­fi­cien­t, to im­age RSS feed­s, re­cur­sive gal­leries, gallery meta­data, im­age texts, and much more.

File Pipelines

Want to min­i­mize your CSS? Tidy your HTM­L? pngcrush your im­ages? ap­ply HTML trans­for­ma­tion­s? Oth­er things I can't imag­ine?

File pipe­lines would bring the pow­er of the unix shell to a site gen­er­a­tor, let­ting you con­nect lego-­like fil­ter­s, some pro­vid­ed, some from the com­mu­ni­ty, in­to a pow­er­ful ma­chin­ery.

Online Editing (Alva)

While stat­ic site gen­er­a­tors have lots of ben­e­fit­s, they have one sig­nif­i­cant down­sid­e: you ed­it the files in your own de­vice. A on­line ed­i­tor for Niko­la lets you ed­it them through a web in­ter­face for blog­ging-from-ay­where good­ness.

Nikola Hosting (Shoreham)

Why not cre­ate a ser­vice where the us­er feeds posts to a serv­er and then the serv­er pub­lish­es them? The feed­ing can be via a DVC­S, or a file sync ser­vice, or via on­line ed­i­tors, and the out­put is pub­lished au­to­mat­i­cal­ly or at the push of a but­ton.

Drafts

I don't do draft­s. I type and that's it. But oth­ers pre­fer more cau­tious and sane ap­proach­es. So, how should drafts work? While the fea­ture may be easy to im­ple­men­t, it's a good be­gin­ner pro­gram­mer's task, where you have to think more about what you want to achieve and pro­vid­ing a good us­er ex­pe­ri­ence than about just bang­ing code.

So, is there some­thing you saw in an­oth­er stat­ic blog gen­er­a­tor and Niko­la lack­s? Any cool ideas and want a friend­ly code­base to hack them on? Do you have any crazy ideas noone would touch with a ten-­foot-pole but you think would be awe­some to have?

Well, now's a good time to talk about it!

There is a large thread in red­dit about us­ing Python as con­fig­u­ra­tion file for­mat. I want to be clear about that:

DON'T DO THAT, UN­LESS YOU HAVE A VERY GOOD REA­SON.

If you need to ask if it's a good idea, then you don't have a good rea­son. If you are sure that you have a good rea­son, then maybe you have a good rea­son.

There are many rea­sons for it, but I will ex­plore just two, then of­fer a sug­ges­tion.

Python is read­-on­ly, and con­fig­ur­ing is not pro­gram­ming.

Sure, it's easy to use python as a con­fig file. You just im­port the thing, and there you go, all the da­ta is there. But now your con­fig­u­ra­tion syn­tax is a gen­er­al pur­pose lan­guage that can do things like pop up di­alogs when you parse it.

Your con­fig can now de­pend on the whole in­ter­net, the hard­ware, the weath­er, and in­ter­ac­tive in­put. Pow­er­ful? Sure. Good idea? May­be, some­times. But your ap­pli­ca­tion is now not able to con­fig­ure it­self.

If your ap­pli­ca­tion wants to store any kind of set­ting, it won't be able to. So most in­ter­ac­tive, desk­top app­s, just should not use python for this, ev­er.

But what about non-in­ter­ac­tive tool­s? Well, us­ing python means that oth­er tools can't write to the con­fig file, ei­ther, which makes the tool less pow­er­ful. The pow­er to have tools use tools is one of the cor­ner­stones of mod­ern com­put­ing, and you just cut your app off that ecosys­tem. De­pend­ing on what lan­guage the tool us­es it may not even be able to parse your con­fig file.

And what hap­pens when some­one is told "use this con­fig frag­ment to achieve X"? Well, odds are, if the re­cip­i­ent has done any­thing that takes ad­van­tage of us­ing python as a con­fig for­mat, then the frag­ment will not work. It would be like do­ing copy­/­paste from ran­dom code in github in­to your own pro­gram and ex­pect­ing it to work.

So, you can't write to it from the ap­p, you can't get con­fig­u­ra­tion tips from the in­ter­net, you can't use oth­er tools to mod­i­fy con­fig files, and oth­er tools have a hard time pars­ing your files.

Al­so, it means that to han­dle the gen­er­al case of con­fig­ur­ing your ap­p, you need a pro­gram­mer. That is al­most cer­tain­ly overkil­l. Very few apps need that kind of thing. If your app can on­ly be con­fig­ured by pro­gram­mer­s, you may have failed at mak­ing a good app (ex­cep­tions ex­ist).

And what's the ad­vice? Well, the ad­vice is "don't do that" and the corol­lary is "con­fig­ure us­ing data, not code". use IN­Is, or XM­L, or YAM­L, or JSON, or plain text files, or what­ev­er. But not code.

PS: My lat­est pro­jec­t, Niko­la us­es python as a con­fig­u­ra­tion lan­guage. I thought I had a good rea­son. I did­n't.

Three years ago, I start­ed a se­ries of long posts called "PyQt by Ex­am­ple". It reached five posts be­fore I aban­doned for a se­ries of rea­sons that don't mat­ter any­more. That se­ries is com­ing back start­ing next week, rewrit­ten, im­proved and ex­tend­ed.

It will do so in a new site, and the "old" posts will be re­tired to an ar­chive page. Why? Well, the tech­nolo­gies used in some of them are ob­so­lete or don't quite work nowa­days. So, the new ver­sions will be the pre­ferred ones.

And while I am not promis­ing any­thing, I have enough writ­ten to make this some­thing quite longer, more nice­ly lay­out­ed, more in­ter­est­ing and make it cov­er more ground. BUT, while do­ing some checks on the traf­fic sta­tis­tics for the old post­s, some things popped out.

This was very popular

About 60% of my site's traf­fic goes to those five post­s. Out of about 1200 posts over 12 years, 60% of the view­ers go to the 0.4% of the pages. That is a lot.

It's a long tail

The traf­fic has not de­creased in three years. If any­thing, it has in­creased

So, all this means there is a de­sire for PyQt doc­u­men­ta­tion that is not sat­is­fied. I am not sur­prised: PyQt is great, and the rec­om­mend­ed book is not free, so there is bound to be a lot of de­mand.

And, here's the not-­so-rosy bit: I had un­ob­tru­sive, rel­e­van­t, out­-of-the-way-but-vis­i­ble ads in those pages for more than two years. Of the 70000 unique vis­i­tors, not even one clicked on an ad. Don't wor­ry, I was not ex­pect­ing to get mon­ey out of them (although I would love to some day col­lect a $100 check in­stead of hav­ing google hold my mon­ey for me ad eter­num).

But re­al­ly? Not even one ad click? In more than two years, thou­sands of peo­ple? I have to won­der if I just at­tract cheap peo­ple ;-)

This fri­day marks the be­gin­ning of the 2012 Py­Cam­p. What's Py­Cam­p? It's a lot of python pro­gram­mers (this year, about 50) gath­er­ing in an iso­lat­ed place for a long week­end, with noth­ing to do ex­cept code. We will have our meals catered, there are no TVs, there is hard­ly any­thing with­in walk­ing dis­tance, and it's go­ing to be very cold.

So, it's go­ing to be awe­some. It's a rare chance for me to spend a few days hack­ing at my own per­son­al pro­ject­s, un­in­ter­rupt­ed by more im­por­tant things like fam­i­ly, work, cook­ing, or so­cial­iz­ing ex­cept be­tween nerd­s.

Sure, there is the oc­ca­sion­al mono­cy­cle rid­ing, or jug­gling lesson, or shoot­ing prac­tice, but re­al­ly, three or four sol­id days of hack­ing.

I in­tend to work on projects re­lat­ed to Niko­la my stat­ic site gen­er­a­tor, so if you are in­ter­est­ed in that and go­ing to py­cam­p, I want to talk with you.

And if you are in­ter­est­ed but not go­ing to py­cam­p, there is no rea­son not to join in a vir­tu­al sprint. We'll have in­ter­net. There is IR­C. I will have time. It's a week­end! Please share any in­ter­est­ing ideas you have about stat­ic site gen­er­a­tors in the niko­la-dis­cuss group and we'll see how much can get im­ple­ment­ed or at least start­ed.

One of the most im­por­tant things when you are build­ing a stat­ic site gen­er­a­tor like Niko­la is that your site should not be bro­ken. So, I re­al­ly should have done this ear­li­er ;-)

This is a very sim­ple link check­er that en­sures the pages Niko­la gen­er­ates have no bro­ken links. I will make it part of Niko­la prop­er once it's more pol­ished and doit sup­ports get­ting a list of tar­gets

To try it, get it and run it from the same place where you have your conf.py, right after you run doit.

import os
import urllib
from urlparse import urlparse

import lxml.html

def analyze(filename):
    try:
        # Use LXML to parse the HTML
        d = lxml.html.fromstring(open(filename).read())
        for l in d.iterlinks():
            # Get the target link
            target = l[0].attrib[l[1]]
            if target == "#":  # These are always valid
                continue
            parsed = urlparse(target)
            # We only handle relative links.
            # TODO: check if the URL points to inside the generated
            # site and check it anyway
            if parsed.scheme:
                continue
            # Ignore the fragment, since the link will still work
            # TODO: check that the fragment is valid
            if parsed.fragment:
                target = target.split('#')[0]
            # Calculate what file or folder this points to
            target_filename = os.path.abspath(
                os.path.join(os.path.dirname(filename), urllib.unquote(target)))
            # Check if it exists, or report it
            if not os.path.exists(target_filename):
                print "In %s broken link: " % filename, target
    except Exception as exc:
        # Something bad happened, report
        print "Error with:", filename, exc

# This is hackish: we use doit to get a list of all
# generated files. Minor modifications would let you check
# the non-generated files as well.

for task in os.popen('doit list --all', 'r').readlines():
    task = task.strip()
    if task.split(':')[0] in (
        'render_tags',
        'render_archive',
        'render_galleries',
        'render_indexes',
        'render_pages',
        'render_site') and '.html' in task:
            # It looks like a generated HTML file
            analyze(task.split(":")[-1])