Posts about open source (old posts, page 6)

2010-10-01 15:12

Making your app modular: Yapsy

That a plugin architecture for a complex app is a good idea is one of those things that most people kinda agree on. One thing we don't quite agree is how the heck are we going to make out app modular?

One way to do it (if you are coding python) is using Yapsy.

Yapsy is awesome. Also, yapsy is a bit underdocumented. Let's see if this post fixes that a bit and leaves just the awesome.

Update: I had not seen the new Yapsy docs, released a few days ago. They are much better than what was there before :-)

Here's the general idea behind yapsy:

  • You create a Plugin Manager that can find and load plugins from a list of places (for example, from ["/usr/share/appname/plugins", "~/.appname/plugins"]).

  • A plugin category is a class.

  • There is a mapping between category names and category classes.

  • A plugin is a module and a metadata file. The module defines a class that inherits from a category class, and belongs to that category.

    The metadata file has stuff like the plugin's name, description, URL, version, etc.

One of the great things about Yapsy is that it doesn't specify too much. A plugin will be just a python object, you can put whatever you want there, or you can narrow it down by specifying the category class.

In fact, the way I have been doing the category classes is:

  • Start with an empty class
  • Implement two plugins of that category
  • If there is a chunk that's much alike in both, move it into the category class.

But trust me, this will all be clearer with an example :-)

I will be doing it with a graphical PyQt app, but Yapsy works just as well for headless of CLI apps.

Let's start with a simple app: an HTML editor with a preview widget.

//ralsina.me/static/yapsy/editor1.jpeg

A simple editor with preview

Here's the code for the app, which is really simple (it doesn't save or do anything, really, it's just an example):

editor1.py

from PyQt4 import QtCore, QtGui, QtWebKit
import os, sys

class Main(QtGui.QWidget):
    def __init__(self):
        QtGui.QWidget.__init__(self)
        self.layout = QtGui.QVBoxLayout()
        self.editor = QtGui.QPlainTextEdit()
        self.preview = QtWebKit.QWebView()
        self.layout.addWidget(self.editor)
        self.layout.addWidget(self.preview)
        self.editor.textChanged.connect(self.updatePreview)
        self.setLayout(self.layout)

    def updatePreview(self):
        self.preview.setHtml(self.editor.toPlainText())

def main():
    # Again, this is boilerplate, it's going to be the same on
    # almost every app you write
    app = QtGui.QApplication(sys.argv)
    window=Main()
    window.show()
    # It's exec_ because exec is a reserved word in Python
    sys.exit(app.exec_())

if __name__ == "__main__":
    main()

Note

From now on listings will not include the main function, because it never changes.

But this application has an obvious limit: you have to type HTML in it. Why not type python code in it and have it convert to HTML for display? Or Wiki markup, or restructured text?

You could, in principle, just implement all those modes, but then you are assuming the responsability of supporting every thing-that-can-be-turned-into-HTML. Your app would be a monolith. That's where yapsy enters the scene.

So, let's create a plugin category, called "Formatter" which takes plain text and returns HTML. Then we add stuff in the UI so the user can choose what formatter he wants, and implement two of those.

Here's our plugin category class:

categories.py

class Formatter(object):
    """Plugins of this class convert plain text to HTML"""

    name = "No Format"

    def formatText(self, text):
        """Takes plain text, returns HTML"""
        return text

Of course what good is a plugin architecture without any plugins for it? So, let's create two plugins.

First: a plugin that takes python code and returns HTML, thanks to pygments.

plugins/pygmentize.py

from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter

from categories import Formatter

class Pygmentizer(Formatter):
    name = "Python Code"

    def formatText(self, text):
        return highlight(text, PythonLexer(), HtmlFormatter(full=True))

See how it goes into a plugins folder? Later on we will tell yapsy to search there for plugins.

To be recognized as a plugin, it needs a metadata file, too:

plugins/pygmentize.yapsy-plugin

[Core]
Name = Python Code
Module = pygmentize

[Documentation]
Author = Roberto Alsina
Version = 0.1
Website = //ralsina.me
Description = Highlights Python Code

And really, that's all there is to making a plugin. Here's another one for comparison, which uses docutils to format reStructured Text:

plugins/rest.py

from categories import Formatter
import docutils.core
import docutils.io


class Rest(Formatter):
    name = "Restructured Text"

    def formatText(self, text):
        output = docutils.core.publish_string(
            text, writer_name = 'html'
        )
        return output

plugins/rest.yapsy-plugin

[Core]
Name = Restructured Text
Module = rest

[Documentation]
Author = Roberto Alsina
Version = 0.1
Website = //ralsina.me
Description = Formats restructured text

And here they are in action:

//ralsina.me/static/yapsy/editor2.jpeg

reSt mode

//ralsina.me/static/yapsy/editor3.jpeg

Python mode

Of course using categories you can do things like a "Tools" category, where the plugins get added to a Tools menu, too.

And here's the application code:

editor2.py

from categories import Formatter
from yapsy.PluginManager import PluginManager

class Main(QtGui.QWidget):
    def __init__(self):
        QtGui.QWidget.__init__(self)
        self.layout = QtGui.QVBoxLayout()
        self.formattersCombo = QtGui.QComboBox()
        self.editor = QtGui.QPlainTextEdit()
        self.preview = QtWebKit.QWebView()

        self.layout.addWidget(self.formattersCombo)
        self.layout.addWidget(self.editor)
        self.layout.addWidget(self.preview)

        self.editor.textChanged.connect(self.updatePreview)
        self.setLayout(self.layout)

        # Create plugin manager
        self.manager = PluginManager(categories_filter={ "Formatters": Formatter})
        self.manager.setPluginPlaces(["plugins"])

        # Load plugins
        self.manager.locatePlugins()
        self.manager.loadPlugins()

        # A do-nothing formatter by default
        self.formattersCombo.addItem("None")
        self.formatters = {}
        print self.manager.getPluginsOfCategory("Formatters")
        for plugin in self.manager.getPluginsOfCategory("Formatters"):
            print  "XXX"
            # plugin.plugin_object is an instance of the plugin
            self.formattersCombo.addItem(plugin.plugin_object.name)
            self.formatters[plugin.plugin_object.name] = plugin.plugin_object

    def updatePreview(self):
        # Check what the current formatter is
        name =  unicode(self.formattersCombo.currentText())
        text = unicode(self.editor.toPlainText())
        if name in self.formatters:
            text = self.formatters[name].formatText(text)
        self.preview.setHtml(text)

In short: this is easy to do, and it leads to fixing your application's internal structure, so it helps you write better code.

Full source code for everything.

2010-09-25 02:41

Quick hack: rss2epub -- it does what it says.

One of my favourite things about Aranduka as a project is that it's an endless source of small, limited side projects.

For example, Aranduka is now close to being able to sync my book collection to my phone. But... what if what I want to read on the train is not a book but, say, a blog?

Well, blogs provide their content via a feed. And A feed is a collection of HTML pieces glued into a structure plus some data like author and such.

And there's a great module for parsing them, called feedparser. And I have written not one, not two, not three, but four RSS aggregators in the past.

So, how about converting the feed into something my phone can handle? [#] Would it be hard to do?

Well... not really hard. It's mostly a matter of taking a small, sample ePub document (created by Calibre) writing a few templates, feeding it the data from feedparser and zipping it up.

For example, this is this blog, as an ePub and here's FBReader reading it:

Share photos on twitter with Twitpic

As usual, the code is open, and it's here in aranduka's mercurial.

It's not really interesting code, and requires templite feedparser and who knows what else.

The produced ePub doesn't validate, and it probably never will, because it has chunks of the original feed in it, so standard compliance doesn't depend on rss2epub.

Also, you get no images. That would imply parsing and fixing all img elements, I suppose, and I am not going to do it right now.

[#] I first saw this feature in plucker a long time ago, and I know Calibre has it too.

2010-09-18 01:55

Introducing Aranduka

Yes, it's yet another program I am working on. But hey, the last few I started are actually pretty functional already!

And... I am not doing this one alone, which should make it more fun.

It's an eBook (or just any book?) manager, that helps you keep your PDF/Mobi/FB2/whatever organized, and should eventually sync them to the device you want to use to read them.

What works now? See the video!

In case that makes no sense to you:

  • You can get books from FeedBooks. Those books will get downloaded, added to your database, tagged, the cover fetched, etc. etc.

  • You can import your current folder of books in bulk.

    Aranduka will use google and other sources to try to guess (from the filename) what book that is and fill in the extra data about it.

  • You can "guess" the extra data.

    By marking certain data (say, the title) as reliable, Aranduka will try to find some possible books that match then you can choose if it's right.

    Of course you can also edit that data manually.

And that's about it. Planned features:

  • Way too many to list.

The goals are clear:

  • It should be beautiful (I know it isn't!)
  • It should be powerful (not yet!)
  • It should be better than the "competition"

If those three goals are not achieved, it's failure. It may be a fun failure, but it would still be a failure.

2010-09-14 12:56

Very pythonic progress dialogs.

Sometimes, you see a piece of code and it just feels right. Here's an example I found when doing my "Import Antigravity" session for PyDay Buenos Aires: the progressbar module.

Here's an example that will teach you enough to use progressbar effectively:

progress = ProgressBar()
for i in progress(range(80)):
    time.sleep(0.01)

Yes, that's it, you will get a nice ASCII progress bar that goes across the terminal, supports resizing and moves as you iterate from 0 to 79.

The progressbar module even lets you do fancier things like ETA or fie transfer speeds, all just as nicely.

Isn't that code just right? You want a progress bar for that loop? Wrap it and you have one! And of course since I am a PyQt programmer, how could I make PyQt have something as right as that?

Here'show the output looks like:

progress

You can do this with every toolkit, and you probably should!. It has one extra feature: you can interrupt the iteration. Here's the (short) code:

# -*- coding: utf-8 -*-
import sys, time
from PyQt4 import QtCore, QtGui

def progress(data, *args):
    it=iter(data)
    widget = QtGui.QProgressDialog(*args+(0,it.__length_hint__()))
    c=0
    for v in it:
        QtCore.QCoreApplication.instance().processEvents()
        if widget.wasCanceled():
            raise StopIteration
        c+=1
        widget.setValue(c)
        yield(v)

if __name__ == "__main__":
    app = QtGui.QApplication(sys.argv)

    # Do something slow
    for x in progress(xrange(50),"Show Progress", "Stop the madness!"):
        time.sleep(.2)

Have fun!

2010-09-10 11:52

The first english Issue of PET (our Python Magazine) is out!

Hell yeah! It has been a lot of work but it's out at http://revista.python.org.ar

Some articles:

  • PyAr, The History
  • from gc import commonsense - Finish Him!
  • Painless Concurrency: The multiprocessing Module
  • Introduction to Unit Testing with Python
  • Taint Mode in Python
  • Applied Dynamism
  • Decorating code (Part 1)
  • Web2Py for Everybody

It's available in pretty much every format anyone can read, and if your favourite is not there, we will make it for you or may I be smote by the flying spaghetti monster's noodly appendage!

AFAIK there is no other Python magazine being published (feel free to correct me), so it's kind of a big thing for us in PyAr (the Argentina Python community) that we are doing one, and in two languages.

But why stop here? Want it to be available in your language? Contact us at [email protected] it may be doable!

And of course, very soon there will be a call for articles for Issue 2, and trust me: that one's going to be epic: this one was just a warmup.

2010-09-09 13:50

Making deployment of desktop Python apps trivial: an idea

Proprietor and printer in front of Schwartz Print Shop in Minneapolis

Here's what I'm thinking: how hard could it be to make PyQt app deployment absolutely easy? Well, I am guessing: not very hard.

Here's the trick: see what works in the real world, and adopt it.

Question: what has deployed billions of apps and has its users happy? Answer: phones app stores.

Question: how do they work? Answer: well, that's not that short, so let's start explaining.

As I see it, a reasonable app store has the following components:

A Stable Deployment Target

You can't deploy from the store if you don't know what you are deploying into. If the target platform is shaky, you just can't know how to deploy without user assistance, and we are trying to make this easy for the user, which means that's not acceptable.

So, what's a stable deployment target we can provide?

  • PyQt (so we can deploy GUIs to all major desktop platforms)
  • Python standard library
  • Selected modules

What can be (and should be) bundled with the app?

  • Pure python modules
  • Artwork and other resources

What may be bundled:

  • Python modules written in C/C++, but you then have to redo the app for each platform, and that kinda sucks.

Deployment Services

  • Apps should be able to check if there is a new version of them in the store, to ask for upgrades.
  • Apps should be added by the deployment platform nicely into the host system's menus, desktop, etc.

Monetization Services

  • Some way to charge for apps. Even for open source apps, you could ask for U$S0.99 if you install them through the store. Optional, of course, and up to the app owner.
  • Ad platform? There must be a good one for desktop apps somewhere?

The Store Itself

  • A website that downloads a "package" associated with a local deployment application.
  • A app store app. Install things not via web, but via a desktop application.

I don't expect a functional version of this would take me more than a week working fulltime to implement. Of course then there are all sorts of usability, looks, etc. things to consider.

And... I am going to do something I very rarely do. I am going to ask for money.

As an experiment, I have setup a project at http://www.indiegogo.com/Qt-Shop and set a funding goal of U$S 600.

There you can fund me. I promise that if the project is totally funded, I will deliver. If it isn't, I may deliver anyway. I would prefer to have the money though.

The platform would be released under GPLv2 or later.

2010-09-08 13:28

Why we are here.

Warning: rant ahead.

Yesterday the government of Argentina announced that they are giving way 3 million netbooks to students. They also announced that they are giving them the option of Ubuntu or Windows 7.

There was, of course, the typical reaction from the FLOSS side: why are they giving Windows to the students when Linux is better? It's unfair that the government pays for Windows!

I am here to tell you to grow up and stop being a baby. I am here to tell you to stop treating others like babies.

I think I can do this because I am immune to criticism from the FLOSS crowd: I am a member of that crowd. I have an awesome FLOSS pedigree, I have used nothing but Linux for over 15 years. And I have a thick skin and I don't care much what other people say, in principle, unless they give me good reasons to care. And I am telling you to stop complaining.

I am telling you that if the only reason to use a specific piece of software is because it's cheaper, you are accepting that piece of software sucks.

I want people not to just use Linux, I want them to want to use Linux. I want them to wait anxiously for the next release of Ubuntu or Firefox or whatever.

And the first step towards excellence is wanting to be excellent. If having to pay nothing for Windows or Ubuntu there is a certainty that Windows will win, then Ubuntu freaking sucks and needs to improve. People are not adopting it even if it's free? Then something is wrong, and figuring out what is important.

But even more important than finding the missing piece is knowing a piece is missing. Open source has grown complacent. It's grown self righteous. It's become adolescent, sure of its awesomeness and immortality.

I don't believe in many things, but I do believe in free will. I believe that people are not morons, I believe that if they prefer Windows, it's because it does something better, and I believe that whatever that is (and I don't really know what it may be), it can be found, and can be improved, and can be replaced, and other things can be added, and people will want to use the better product.

And if they don't... well, at least we fought an honest fight, and we did our best, and we (hopefully) had fun in the process, and pushed the envelope, and created nice things, and the users are better off in the end even if our babies are not the chosen ones, because we raised the level of everything.

For example, before Linux, Windows sucked much, much more than it does now, and I think many of those improvements were because of Linux, and I am happy that today Windows users have a OS that doesn't stink.

I want free and open source software to be used because it's awesome, not because it's cheap. Awesome and cheap I can live with. Just cheap? That sucks.

And the constant "they use windows because they don't know better"? That's patronizing and condescending, and very, very annoying. And if it annoys me, who is not the target of the lame condescension, trust me, it annoys the crap out of Windows users.

Grow a spine, get your asses into gear, start making awesome stuff, kick ass with quality. That's why we are here. Not to be the cheapest date in town.

2010-09-01 16:36

Goodreads+webcam+python+zbar == hackfun!

I am a big fan of GoodReads a social network for people who read books.

I read a lot, and I like that I can see what other people think before starting a book, and I can put my short reviews, and I can see what I have been reading, and lots more.

In fact, goodreads is going to be a big part of a project I am starting with some PyAr guys.

One thing I have been lazy about is adding my book list to goodreads, because it's a bit of a chore.

Well, chore no more!

Here's how to do it, the hacker way...

  1. Get zbar
  2. Get a cheap webcam
  3. Get a book
  4. Get a 7-line python program (included below)

Now watch the video...

Cute, isn't it?

Here's the code:

import os

p=os.popen('/usr/bin/zbarcam','r')
while True:
    code = p.readline()
    print 'Got barcode:', code
    isbn = code.split(':')[1]
    os.system('chromium http://www.goodreads.com/search/search?q=%s'%isbn)

Contents © 2000-2018 Roberto Alsina