Posts about programming (old posts, page 37)

2012-07-02 22:17

The Future of PyQt by Example

Three years ago, I started a series of long posts called "PyQt by Example". It reached five posts before I abandoned for a series of reasons that don't matter anymore. That series is coming back starting next week, rewritten, improved and extended.

It will do so in a new site, and the "old" posts will be retired to an archive page. Why? Well, the technologies used in some of them are obsolete or don't quite work nowadays. So, the new versions will be the preferred ones.

And while I am not promising anything, I have enough written to make this something quite longer, more nicely layouted, more interesting and make it cover more ground. BUT, while doing some checks on the traffic statistics for the old posts, some things popped out.

This was very popular
About 60% of my site's traffic goes to those five posts. Out of about 1200 posts over 12 years, 60% of the viewers go to the 0.4% of the pages. That is a lot.
It's a long tail
The traffic has not decreased in three years. If anything, it has increased
https://p.twimg.com/Aw0MHhoCAAAXmro.png:large

A long and tall tail.

So, all this means there is a desire for PyQt documentation that is not satisfied. I am not surprised: PyQt is great, and the recommended book is not free, so there is bound to be a lot of demand.

And, here's the not-so-rosy bit: I had unobtrusive, relevant, out-of-the-way-but-visible ads in those pages for more than two years. Of the 70000 unique visitors, not even one clicked on an ad. Don't worry, I was not expecting to get money out of them (although I would love to some day collect a $100 check instead of having google hold my money for me ad eternum).

But really? Not even one ad click? In more than two years, thousands of people? I have to wonder if I just attract cheap people ;-)

2012-05-16 17:23

Hack English Instead

Lots of noise recently about Jeff Atwood's post about why you should not learn to code. I am here now telling you you should learn to code. But only after you learn a few other things.

You should learn to speak. You should learn to write. You should learn to listen. You should learn to read. You should learn to express yourself.

Richard Feynman once described his problem solving algorithm as follows:

  1. Write down the problem
  2. Think real hard
  3. Write down the solution

Most of us cannot do that because we are not Richard Feynman and thus, sadly, cannot keep all the solution in our head in step 2, so we need to iterate a few times, thinking (not as hard as he could) and writing down a bit of the solution on each loop.

And while we who code are unusually proud of our ability to write down solutions in such a clear and unforgiving way that even a computer can follow them, it's ten, maybe a hundred times more useful to know how to write it down, or say it, in such a way that a human being can understand it.

Explanations fit for computers are bad for humans and viceversa. Humans accept much more compact, ambiguous, and expressive code. You can transfer high level concepts or design to humans much easier than to computers, but algorithms to computers much easier than to humans.

I have a distrust of people who are able to communicate to computers easier than with fellow humans, a suspicion that they simply have a hole in their skillset, which they could easily fix if they saw it as essential.

And it is an essential skill. Programmers not only run on coffee and sugar and sushi and doritos, they run on happiness. They have a finite endowment of happiness and they spend it continuously, like drunken sailors. They perform an activity where jokingly they measure productivity on curses per hour, a lonely endeavour that isolates them (us) from other humans, from family and friends.

If a developer cannot communicate he isolates. When he isolates he can't cooperate, he cannot delegate. He can't give ideas to others, he can't receive them, he can't share.

And since lots of our communication is via email, and chat, and bug reports, and blogs, it's better if he can write. A developer who cannot write is at a serious disadvantage. A developer who cannot write to express an idea cannot explain, he doesn't make his fellows better. He's a knowledge black hole, where information goes to die behind the event horizon of his skull.

So, learn to write. Learn to speak. Learn to read and listen. Then learn to code.

2012-04-27 18:52

Shoreham: Blogging with Ubuntu One (a teaser)

So, today, I ripped off a great service offered by http://calepin.co and implemented a prototype blog-through-Ubuntu-One web application. Of course, it's powered by Nikola,

The code is absolute nonsense, and it needs to be looked at by someone who understands Django, OAuth, OpenID, and programming in general better than I do, but hey, it does work (for a very loose definition of "work").

It's called Shoreham and no, you can't have it yet.

As a teaser, here's a video. With a pony.

In the near future I will do a better post about this explaining the code, etc.

2012-04-24 22:46

Alva

Alva is almost the opposite of Nikola. If Nikola is about making static sites, Alva is a dynamic site. However, as Hegel suggests, from the thesis and the antithesis comes the synthesis.

So, Alva is about dynamically creating static sites. If you want to have Nikola in your server instead of in your own computer, and have the convenience of an online tool, that's the niche Alva tries to fill.

So, you would install Alva, and use it like any other web-based blogging tool. Yet, behind the scenes, you would have Nikola, and all the performance and security benefits of static sites.

And maybe someday, I (or someone) will put up a multi-user version of Alva, and you will be able to get hosted blogs, knowing all the data is yours and you can leave anytime and do your own thing.

This is very very early stages. So early it does not work yet. But here's a teaser:

//ralsina.me/galleries/random/nikola-admin3.thumbnail.png

There is no firm timeframe for this, it depends on a ton of other stuff and may not even happen.

2012-04-22 21:58

Nikola Screencast

I did some work today to get Nikola properly packaged. This involves some minor changes on the workflow for site authors. I am not 100% sure I have it right yet, so here is a short video showing how it works right now in the packaging branch I am doing.

The new thing is the nikola init foldername command, the rest is all old stuff. Basically, you stop having a full copy of Nikola for each site and everything is in a centralized location.

You can still do your own themes by putting them in themes/themename and add new tasks, files, etc. The configuration dodo.py is unchanged except for the "magic bit" which is slightly different.

So, not really invasive, easy to migrate to, and enables much easier updates in the future, as long as we don't break any important stuff in a non-compatible way.

Here is the video:

2012-04-19 23:34

Jinja support in Nikola ready for testing

Since some people really don't like Mako, and prefer Jinja2, I decided to make the template engine modular in Nikola my static site/blog generator.

All things considered, there is very little difference on the API side:

//ralsina.me/galleries/random/jinjavsmako.thumbnail.png

I even ported the default theme to Jinja! And not all that much is different on the template side, either:

//ralsina.me/galleries/random/jinjavsmako1.thumbnail.png

But hey, to each his own. Other template engines are probably easy to plug, too.

Branch up for review here in github and yes, anyone reading this who knows Jinja2 or python is welcome to review it.

2012-04-16 23:12

Smiljan, a Small Planet Generator

I maintain a couple of small "planet" sites. If you are not familiar with planets, they are sites that aggregate RSS/Atom feeds for a group of people related somehow. It makes for a nice, single, thematic feed.

Recently, when changing them from one server to another, everything broke. Old posts were new, feeds that had not been updated in 2 years were always with all its posts on top... a disaster.

I could have gone to the old server, and started debugging why rawdog was doing that, or switch to planet, or look for other software, or use an online aggregator.

Instead, I started thinking... I had written a few RSS aggregators in the past... Feedparser is again under active development... rawdog and planet seem to be pretty much abandoned... how hard could it be to implement the minimal planet software?

Well, not all that hard, that's how hard it was. Like it took me 4 hours, and was not even difficult.

One reason why this was easier than what planet and rawdog achieved is that I am not doing a static site generator, because I already have one so all I need this program (I called it Smiljan) to do is:

  • Parse a list of feeds and store it in a database if needed.
  • Download those feeds (respecting etag and modified-since).
  • Parse those feeds looking for entries (feedparser does that).
  • Load those entries (or rather, a tiny subset of their data) in the database.
  • Use the entries to generate a set of files to feed Nikola
  • Use nikola to generate and deploy the site.

So, here is the final result: http://planeta.python.org.ar which still needs theming and a lot of other stuff, but works.

I implemented Smiljan as 3 doit tasks, which makes it very easy to integrate with Nikola (if you know Nikola: add "from smiljan import *" in your dodo.py and a feeds file with the feed list in rawdog format) and voilá, running this updates the planet:

doit load_feeds update_feeds generate_posts render_site deploy

Here is the code for smiljan.py, currently at the "gross hack that kinda works" stage. Enjoy!

# -*- coding: utf-8 -*-
import codecs
import datetime
import glob
import os
import sys

from doit.tools import timeout
import feedparser
import peewee


class Feed(peewee.Model):
    name = peewee.CharField()
    url = peewee.CharField(max_length = 200)
    last_status = peewee.CharField()
    etag = peewee.CharField(max_length = 200)
    last_modified = peewee.DateTimeField()

class Entry(peewee.Model):
    date = peewee.DateTimeField()
    feed = peewee.ForeignKeyField(Feed)
    content = peewee.TextField(max_length = 20000)
    link = peewee.CharField(max_length = 200)
    title = peewee.CharField(max_length = 200)
    guid = peewee.CharField(max_length = 200)

Feed.create_table(fail_silently=True)
Entry.create_table(fail_silently=True)

def task_load_feeds():
    feeds = []
    feed = name = None
    for line in open('feeds'):
        line = line.strip()
        if line.startswith('feed'):
            feed = line.split(' ')[2]
        if line.startswith('define_name'):
            name = ' '.join(line.split(' ')[1:])
        if feed and name:
            feeds.append([feed, name])
            feed = name = None

    def add_feed(name, url):
        f = Feed.create(
            name=name,
            url=url,
            etag='caca',
            last_modified=datetime.datetime(1970,1,1),
            )
        f.save()

    def update_feed_url(feed, url):
        feed.url = url
        feed.save()

    for feed, name in feeds:
        f = Feed.select().where(name=name)
        if not list(f):
            yield {
                'name': name,
                'actions': ((add_feed,(name, feed)),),
                'file_dep': ['feeds'],
                }
        elif list(f)[0].url != feed:
            yield {
                'name': 'updating:'+name,
                'actions': ((update_feed_url,(list(f)[0], feed)),),
                }


def task_update_feeds():
    def update_feed(feed):
        modified = feed.last_modified.timetuple()
        etag = feed.etag
        parsed = feedparser.parse(feed.url,
            etag=etag,
            modified=modified
        )
        try:
            feed.last_status = str(parsed.status)
        except:  # Probably a timeout
            # TODO: log failure
            return
        if parsed.feed.get('title'):
            print parsed.feed.title
        else:
            print feed.url
        feed.etag = parsed.get('etag', 'caca')
        modified = tuple(parsed.get('date_parsed', (1970,1,1)))[:6]
        print "==========>", modified
        modified = datetime.datetime(*modified)
        feed.last_modified = modified
        feed.save()
        # No point in adding items from missinfg feeds
        if parsed.status > 400:
            # TODO log failure
            return
        for entry_data in parsed.entries:
            print "========================================="
            date = entry_data.get('updated_parsed', None)
            if date is None:
                date = entry_data.get('published_parsed', None)
            if date is None:
                print "Can't parse date from:"
                print entry_data
                return False
            date = datetime.datetime(*(date[:6]))
            title = "%s: %s" %(feed.name, entry_data.get('title', 'Sin título'))
            content = entry_data.get('description',
                    entry_data.get('summary', 'Sin contenido'))
            guid = entry_data.get('guid', entry_data.link)
            link = entry_data.link
            print repr([date, title])
            entry = Entry.get_or_create(
                date = date,
                title = title,
                content = content,
                guid=guid,
                feed=feed,
                link=link,
            )
            entry.save()
    for feed in Feed.select():
        yield {
            'name': feed.name.encode('utf8'),
            'actions': [(update_feed,(feed,))],
            'uptodate': [timeout(datetime.timedelta(minutes=20))],
            }

def task_generate_posts():

    def generate_post(entry):
        meta_path = os.path.join('posts',str(entry.id)+'.meta')
        post_path = os.path.join('posts',str(entry.id)+'.txt')
        with codecs.open(meta_path, 'wb+', 'utf8') as fd:
            fd.write(u'%s\n' % entry.title.replace('\n', ' '))
            fd.write(u'%s\n' % entry.id)
            fd.write(u'%s\n' % entry.date.strftime('%Y/%m/%d %H:%M'))
            fd.write(u'\n')
            fd.write(u'%s\n' % entry.link)
        with codecs.open(post_path, 'wb+', 'utf8') as fd:
            fd.write(u'.. raw:: html\n\n')
            content = entry.content
            if not content:
                content = u'Sin contenido'
            for line in content.splitlines():
                fd.write(u'    %s\n' % line)

    for entry in Entry.select().order_by(('date', 'desc')):
        yield {
            'name': entry.id,
            'actions': [(generate_post, (entry,))],
            }

2012-04-14 11:33

Nikola 2.1.1 + GitHub

By popular request, Nikola now has its source code at GitHub.

Also, if you tried version 2.1 and it failed, try 2.1.1, because I forgot to add a couple of files in one of the themes in 2.1.

2012-04-09 21:41

Nikola 1.2 is out!

Version 1.2 of Nikola, my static site generator and the software behind this very site, is out!

Why build static sites? Because they are light in resources, they are future-proof, because it's easy, because they are safe, and because you avoid lockin.

New Features:

  • Image gallery (just drop pics on a folder)
  • Built-in webserver for previews (doit -a serve)
  • Helper commands to create new posts (doit -a new_post)
  • Google Sitemap support
  • A Handbook!
  • Full demo site included
  • Support for automatic deployment (doit -a deploy)
  • Client-side redirections

And of course the old features:

  • Write your posts in reStructured text
  • Clean, customizable page design (via bootstrap)
  • Comments via Disqus
  • Support any analytics you want
  • Build blogs with tags, feeds, feeds for your tags, indexes, and more
  • Works like a simple CMS for things outside your blog
  • Clean customizable templates using Mako
  • Pure python, and not a lot of it (about 600 lines)
  • Smart builds (doit only rebuilds changed pages)
  • Easy to extend and improve
  • Code displayed with syntax highlighting

Right now Nikola does literally everything I need, so if you try it and need something else... it's a good time to ask!

More info at http://nikola-generator.googlecode.com

2012-03-30 22:59

Nikola 1.1 is out!

A simple yet powerful and flexible static website and blog generator, based on doit, mako, docutils and bootstrap.

I built this to power this very site you are reading, but decided it may be useful to others. The main goals of Nikola are:

  • Small codebase: because I don't want to maintain a big thing for my blog
  • Fast page generation: Adding a post should not take more that 5 seconds to build.
  • Static output: Deployment using rsync is smooth.
  • Flexible page generation: you can decide where everything goes in the final site.
  • Powerful templates: Uses Mako
  • Clean markup for posts: Uses Docutils
  • Don't do stupid builds: Uses doit
  • Clean HTML output by default: Uses bootstrap
  • Comments out of the box: Uses Disqus
  • Tags, with their own RSS feeds
  • Easy way to do a blog
  • Static pages outside the blog
  • Multilingual blog support (my own blog is english + spanish)

I think this initial version achieves all of those goals, but of course, it can be improved. Feedback is very welcome!

Nikola's home page is currently http://nikola-generator.googlecode.com

Contents © 2000-2018 Roberto Alsina