Posts about pyqt

Qt Mac Tips

My team has been working on porting some PyQt stuff to Mac OSX, and we have run into several Qt bugs, sadly. Here are two, and the workarounds we found.

Native dialogs are broken.

Using QFileDialog.getExistingDirectory we noticed the following symptoms:

  • If you do nothing, the dialog went away on its own after about 20 seconds.
  • After you used it once, it may pop up and disappear immediately. Or not.

Solution: use the DontUseNativeDialog option.

Widgets in QTreeWidgetItems don't scroll.

When you use Widgets inside the items of a QTreeWidget (which I know, is not a common case, but hey, it happens), the widgets don't scroll with the items.

Solution: use the -graphicssystem raster options. You can even inject them into argv if the platform is darwin.

The Future of PyQt by Example

Three years ago, I started a series of long posts called "PyQt by Example". It reached five posts before I abandoned for a series of reasons that don't matter anymore. That series is coming back starting next week, rewritten, improved and extended.

It will do so in a new site, and the "old" posts will be retired to an archive page. Why? Well, the technologies used in some of them are obsolete or don't quite work nowadays. So, the new versions will be the preferred ones.

And while I am not promising anything, I have enough written to make this something quite longer, more nicely layouted, more interesting and make it cover more ground. BUT, while doing some checks on the traffic statistics for the old posts, some things popped out.

This was very popular
About 60% of my site's traffic goes to those five posts. Out of about 1200 posts over 12 years, 60% of the viewers go to the 0.4% of the pages. That is a lot.
It's a long tail
The traffic has not decreased in three years. If anything, it has increased
https://p.twimg.com/Aw0MHhoCAAAXmro.png:large

A long and tall tail.

So, all this means there is a desire for PyQt documentation that is not satisfied. I am not surprised: PyQt is great, and the recommended book is not free, so there is bound to be a lot of demand.

And, here's the not-so-rosy bit: I had unobtrusive, relevant, out-of-the-way-but-visible ads in those pages for more than two years. Of the 70000 unique visitors, not even one clicked on an ad. Don't worry, I was not expecting to get money out of them (although I would love to some day collect a $100 check instead of having google hold my money for me ad eternum).

But really? Not even one ad click? In more than two years, thousands of people? I have to wonder if I just attract cheap people ;-)

Ubuntu One APIs by Example (part 1)

One of the nice things about working at Canonical is that we produce open source software. I, specifically, work in the team that does the desktop clients for Ubuntu One which is a really cool job, and a really cool piece of software. However, one thing not enough people know, is that we offer damn nice APIs for developers. We have to, since all our client code is open source, so we need those APIs for ourselves.

So, here is a small tutorial about using some of those APIs. I did it using Python and PyQt for several reasons:

  • Both are great tools for prototyping
  • Both have good support for the required stuff (DBus, HTTP, OAuth)
  • It's what I know and enjoy. Since I did this code on a sunday, I am not going to use other things.

Having said that, there is nothing python-specific or Qt-specific in the code. Where I do a HTTP request using QtNetwork, you are free to use libsoup, or whatever.

So, on to the nuts and bolts. The main pieces of Ubuntu One, from a infrastructure perspective, are Ubuntu SSO Client, that handles user registration and login, and SyncDaemon, which handles file synchronization.

To interact with them, on Linux, they offer DBus interfaces. So, for example, this is a fragment of code showing a way to get the Ubuntu One credentials (this would normally be part of an object's __init__):

# Get the session bus
bus = dbus.SessionBus()

:
:
:

# Get the credentials proxy and interface
self.creds_proxy = bus.get_object("com.ubuntuone.Credentials",
                        "/credentials",
                        follow_name_owner_changes=True)

# Connect to signals so you get a call when something
# credential-related happens
self.creds_iface = dbus.Interface(self.creds_proxy,
    "com.ubuntuone.CredentialsManagement")
self.creds_proxy.connect_to_signal('CredentialsFound',
    self.creds_found)
self.creds_proxy.connect_to_signal('CredentialsNotFound',
    self.creds_not_found)
self.creds_proxy.connect_to_signal('CredentialsError',
    self.creds_error)

# Call for credentials
self._credentials = None
self.get_credentials()

You may have noticed that get_credentials doesn't actually return the credentials. What it does is, it tells SyncDaemon to fetch the credentials, and then, when/if they are there, one of the signals will be emitted, and one of the connected methods will be called. This is nice, because it means you don't have to worry about your app blocking while SyncDaemon is doing all this.

But what's in those methods we used? Not much, really!

def get_credentials(self):
    # Do we have them already? If not, get'em
    if not self._credentials:
        self.creds_proxy.find_credentials()
    # Return what we've got, could be None
    return self._credentials

def creds_found(self, data):
    # Received credentials, save them.
    print "creds_found", data
    self._credentials = data
    # Don't worry about get_quota yet ;-)
    if not self._quota_info:
        self.get_quota()

def creds_not_found(self, data):
    # No credentials, remove old ones.
    print "creds_not_found", data
    self._credentials = None

def creds_error(self, data):
    # No credentials, remove old ones.
    print "creds_error", data
    self._credentials = None

So, basically, self._credentials will hold a set of credentials, or None. Congratulations, we are now logged into Ubuntu One, so to speak.

So, let's do something useful! How about asking for how much free space there is in the account? For that, we can't use the local APIs, we have to connect to the servers, who are, after all, the ones who decide if you are over quota or not.

Access is controlled via OAuth. So, to access the API, we need to sign our requests. Here is how it's done. It's not particularly enlightening, and I did not write it, I just use it:

def sign_uri(self, uri, parameters=None):
    # Without credentials, return unsigned URL
    if not self._credentials:
        return uri
    if isinstance(uri, unicode):
        uri = bytes(iri2uri(uri))
    print "uri:", uri
    method = "GET"
    credentials = self._credentials
    consumer = oauth.OAuthConsumer(credentials["consumer_key"],
                                   credentials["consumer_secret"])
    token = oauth.OAuthToken(credentials["token"],
                             credentials["token_secret"])
    if not parameters:
        _, _, _, _, query, _ = urlparse(uri)
        parameters = dict(cgi.parse_qsl(query))
    request = oauth.OAuthRequest.from_consumer_and_token(
                                        http_url=uri,
                                        http_method=method,
                                        parameters=parameters,
                                        oauth_consumer=consumer,
                                        token=token)
    sig_method = oauth.OAuthSignatureMethod_HMAC_SHA1()
    request.sign_request(sig_method, consumer, token)
    print "SIGNED:", repr(request.to_url())
    return request.to_url()

And how do we ask for the quota usage? By accessing the https://one.ubuntu.com/api/quota/ entry point with the proper authorization, we would get a JSON dictionary with total and used space. So, here's a simple way to do it:

    # This is on __init__
    self.nam = QtNetwork.QNetworkAccessManager(self,
        finished=self.reply_finished)

:
:
:

def get_quota(self):
    """Launch quota info request."""
    uri = self.sign_uri(QUOTA_API)
    url = QtCore.QUrl()
    url.setEncodedUrl(uri)
    self.nam.get(QtNetwork.QNetworkRequest(url))

Again, see how get_quota doesn't return the quota? What happens is that get_quota will launch a HTTP request to the Ubuntu One servers, which will, eventually, reply with the data. You don't want your app to block while you do that. So, QNetworkAccessManager will call self.reply_finished when it gets the response:

def reply_finished(self, reply):
    if unicode(reply.url().path()) == u'/api/quota/':
        # Handle quota responses
        self._quota_info = json.loads(unicode(reply.readAll()))
        print "Got quota: ", self._quota_info
        # Again, don't worry about update_menu yet ;-)
        self.update_menu()

What else would be nice to have? How about getting a call whenever the status of syncdaemon changes? For example, when sync is up to date, or when you get disconnected? Again, those are DBus signals we are connecting in our __init__:

self.status_proxy = bus.get_object(
    'com.ubuntuone.SyncDaemon', '/status')
self.status_iface = dbus.Interface(self.status_proxy,
    dbus_interface='com.ubuntuone.SyncDaemon.Status')
self.status_iface.connect_to_signal(
    'StatusChanged', self.status_changed)

# Get the status as of right now
self._last_status = self.process_status(
    self.status_proxy.current_status())

And what's status_changed?

def status_changed(self, status):
    print "New status:", status
    self._last_status = self.process_status(status)
    self.update_menu()

The process_status function is boring code to convert the info from syncdaemon's status into a human-readable thing like "Sync is up-to-date". So we store that in self._last_status and update the menu.

What menu? Well, a QSystemTrayIcon's context menu! What you have read are the main pieces you need to create something useful: a Ubuntu One tray app you can use in KDE, XFCE or openbox. Or, if you are on unity and install sni-qt, a Ubuntu One app indicator!

http://ubuntuone.com/7iXTbysoMM9PIUS9Ai4TNn

My Ubuntu One indicator in action.

You can find the source code for the whole example app at my u1-toys project in launchpad and here is the full source code (missing some icon resources, just get the repo)

Coming soon(ish), more example apps, and cool things to do with our APIs!

PyQt Quickie: command line parsing

So, you are writing a PyQt app, and you want it to support command line arguments. So you do something like this:

opt_parser = OptionParser()
opt_parser.add_option("-q", dest="quickly", action="store_true",
    help="Do it quickly (default=False)")
(options, args) = opt_parser.parse_args(sys.argv)
app = QApplication(sys.argv)
:
:
:

Or maybe even QApplication([]). Ok, you are doing it wrong. And this is wrong in most tutorials, too. Why? Because Qt (and thus PyQt) supports a bunch of useful command line options already. So if you do it like in the first listing, and pass "-style=oxygen" or whatever, one of the following will happen.

  1. OptParser is going to tell you it's not a valid option and abort
  2. You will ignore the option and not do anything useful with it
  3. You will have your own -style option and do two things with it

All three outcomes are less than ideal.

The right way to do this is:

opt_parser = OptionParser()
opt_parser.add_option("-q", dest="quickly", action="store_true",
    help="Do it quickly (default=False)")
app = QApplication(sys.argv)
(options, args) = opt_parser.parse_args(app.arguments())
:
:
:

This way, you give PyQt a chance to process the options it recognizes, and, then, you get to handle the rest, because app.arguments() has all Qt options removed.

The bad side of this is, you will make --help slightly slower, since it will have to build a QApplication to do nothing, and you will have undocumented options. Solution for both problems left as an exercise.

To write, and to write what.

Some of you may know I have written about 30% of a book, called "Python No Muerde", available at http://nomuerde.netmanagers.com.ar (in spanish only).That book has stagnated for a long time.

On the other hand, I wrote a very popular series of posts, called PyQt by Example, which has (you guessed it) stagnated for a long time.

The main problem with the book was that I tried to cover way too much ground. When complete, it would be a 500 page book, and that would involve writing half a dozen example apps, some of them in areas I am no expert.

The main problem with the post series is that the example is lame (a TODO app!) and expanding it is boring.

¡So, what better way to fix both things at once, than to merge them!

I will leave Python No Muerde as it is, and will do a new book, called PyQt No Muerde. It will keep the tone and language of Python No Muerde, and will even share some chapters, but will focus on developing a PyQt app or two, instead of the much more ambitious goals of Python No Muerde. It will be about 200 pages.

I have acquired permission from my superiors (my wife) to work on this project a couple of hours a day, in the early morning. So, it may move forward, or it may not. This is, as usual, an experiment, not a promise.

PyQt Quickie: Don't Get Garbage Collected

There is one area where Qt and Python (and in consequence PyQt) have major disagreements. That area is memory management.

While Qt has its own mechanisms to handle object allocation and disposal (the hierarchical QObject trees, smart pointers, etc.), PyQt runs on Python, so it has garbage collection.

Let's consider a simple example:

from PyQt4 import QtCore

def finished():
    print "The process is done!"
    # Quit the app
    QtCore.QCoreApplication.instance().quit()

def launch_process():
    # Do something asynchronously
    proc = QtCore.QProcess()
    proc.start("/bin/sleep 3")
    # After it finishes, call finished
    proc.finished.connect(finished)

def main():
    app = QtCore.QCoreApplication([])
    # Launch the process
    launch_process()
    app.exec_()

main()

If you run this, this is what will happen:

QProcess: Destroyed while process is still running.
The process is done!

Plus, the script never ends. Fun! The problem is that proc is being deleted at the end of launch_process because there are no more references to it.

Here is a better way to do it:

from PyQt4 import QtCore

processes = set([])

def finished():
    print "The process is done!"
    # Quit the app
    QtCore.QCoreApplication.instance().quit()

def launch_process():
    # Do something asynchronously
    proc = QtCore.QProcess()
    processes.add(proc)
    proc.start("/bin/sleep 3")
    # After it finishes, call finished
    proc.finished.connect(finished)

def main():
    app = QtCore.QCoreApplication([])
    # Launch the process
    launch_process()
    app.exec_()

main()

Here, we add a global processes set and add proc there so we always keep a reference to it. Now, the program works as intended. However, it still has an issue: we are leaking QProcess objects.

While in this case the leak is very short-lived, since we are ending the program right after the process ends, in a real program this is not a good idea.

So, we would need to add a way to remove proc from processes in finished. This is not as easy as it may seem. Here is an idea that will not work as you expect:

def launch_process():
    # Do something asynchronously
    proc = QtCore.QProcess()
    processes.add(proc)
    proc.start("/bin/sleep 3")
    # Remove the process from the global set when done
    proc.finished.connect(lambda: processes.remove(proc))
    # After it finishes, call finished
    proc.finished.connect(finished)

In this version, we will still leak proc, even though processes is empty! Why? Because we are keeping a reference to proc in the lambda!

I don't really have a good answer for that that doesn't involve turning everything into members of a QObject and using sender to figure out what process is ending, or using QSignalMapper. That version is left as an exercise.

PyQt Quickie: QTimer

QTimer is a fairly simple class: you use it when you want something to happen "in a while" or "every once in a while".

The first case is something like this:

# call f() in 3 seconds
QTimer.singleShot(3000, f)

The second is this:

# Create a QTimer
timer = QTimer()
# Connect it to f
timer.timeout.connect(f)
# Call f() every 5 seconds
timer.start(5000)

Simple, right? Well, yes, but it has some tricks.

  1. You have to keep a reference to``timer``

    If you don't, it willget garbage-collected, and f() will never be called.

  2. It may not call f() in 5 seconds.

    It will call f() more or less 5 seconds after you enter the event loop. That may not be quickly after you start the timer at all!

  3. You may get overlapping calls.

    If f() takes long to finish and re-enters the event loop (for example, by calling processEvents) maybe the timer will timeout and call it again before it's finished. That's almost never a good thing.

So, you can do this:

def f():
    try:
        # Do things
    finally:
        QTimer.singleShot(5000, f)

f()

What that snippet does, is, calls f() only once. But f itself schedules itself to run in 5 seconds. Since it does it in a finally, it will do so even if things break.

That means no overlapping calls. It also means it won't be called every 5 seconds, but 5 seconds plus whatever f takes to run. Also, no need to keep any reference to a QTimer.

Final tip: You can also use QTimer to do something "as soon as you are in the event loop"

QTimer.singleShot(0, f)

Hope it was useful!

Shipping your PyQt app for windows

I have written about this in the past, with the general conclusion being "it's a pain in the ass".

So, now, here is how it's done.

  1. Start with a working PyQt application. In this example, I will use devicenzo.py mostly because:
    1. It is a working PyQt application.
    2. It uses a big chunk of PyQt
    3. It's easy to test
  2. Now you need a setup.py. Here's one that works, with extensive commments.
# We will be using py2exe to build the binaries.
# You may use other tools, but I know this one.

from distutils.core import setup
import py2exe

# Now you need to pass arguments to setup
# windows is a list of scripts that have their own UI and
# thus don't need to run in a console.

setup(windows=['devicenzo.py'],
      options={

# And now, configure py2exe by passing more options;

          'py2exe': {

# This is magic: if you don't add these, your .exe may
# or may not work on older/newer versions of windows.

              "dll_excludes": [
                  "MSVCP90.dll",
                  "MSWSOCK.dll",
                  "mswsock.dll",
                  "powrprof.dll",
                  ],

# Py2exe will not figure out that you need these on its own.
# You may need one, the other, or both.

              'includes': [
                  'sip',
                  'PyQt4.QtNetwork',
                  ],

# Optional: make one big exe with everything in it, or
# a folder with many things in it. Your choice
#             'bundle_files': 1,
          }
      },

# Qt's dynamically loaded plugins and py2exe really don't
# get along.

data_files = [
            ('phonon_backend', [
                'C:\Python27\Lib\site-packages\PyQt4\plugins\phonon_backend\phonon_ds94.dll'
                ]),
            ('imageplugins', [
            'c:\Python27\lib\site-packages\PyQt4\plugins\imageformats\qgif4.dll',
            'c:\Python27\lib\site-packages\PyQt4\plugins\imageformats\qjpeg4.dll',
            'c:\Python27\lib\site-packages\PyQt4\plugins\imageformats\qsvg4.dll',
            ]),
],

# If you choose the bundle above, you may want to use this, too.
#     zipfile=None,
)
  1. Run python setup.py py2exe and get a dist folder full of binary goodness.

And that's it. Except of course, that's not it.

What this will do is create a binary set, either a folder full of things, or a single EXE file. And that's not enough. You have to consider at least the following:

  1. Put everything in resource files: images, qss files, icons, etc. Every file your app needs? Put it in a resource file and load it from there. That way you don't have to care about them if you go the "one exe" road.
  2. Compile .ui files to .py (same reason)
  3. Figure out if you use Qt's plugins, and make them work. This includes: using Phonon, using QtSQL, and using any image formats other than PNG.

After you have that, are you done? NO!

Your windows user will want an installer. I am not going to go into details, but I had a good time using BitRock's InstallBuilder for Qt. It's a nice tool, and it works. That's a lot in this field.

But is that all? NO!

You have to take care of the Visual Studio Runtime. My suggestion? Get a copy of the 1.1MB vcredist_x86.exe (not the larger one, the 1.1MB one), and either tell people to install it manually, or add it to your installer. You are legally allowed (AFAIK) to redistribute that thing as a whole. But not what's in it (unless you have a VS license).

And we are done? NO!

Once you run your app "installed", if it ever prints anything to stderr, you will get either a dialog telling you it did, or worse (if you are in aything newer than XP), a dialog telling you it can't write to a log file, and the app will never work again.

This is because py2exe catches stderr and tries to save it on a logfile. Which it tries to create in the same folder as the binary. Which is usually not allowed because of permissions.

Solution? Your app should never write to stderr. Write an excepthook and catch that. And then remove stderr or replace it with a log file, or something. Just don't let py2exe do it, because the way py2exe does it is broken.

And is that it?

Well, basically yes. Of course you should get 4 or 5 different versions of windows to test it on, but you are pretty much free to ship your app as you wish. Oh, mind you, don't upload it to downloads.com because they will wrap your installer in a larger one that installs bloatware and crap.

So, there you go.