Posts about sysadmin

With iterpipes, python is ready to replace bash for scripting. Really.

This has been a pet peeve of mine for years: programming shell scripts suck. They are ugly and error prone. The only reason why we still do it? There is no real replacement.

Or at least that was the case, until today I met iterpipes at python.reddit.com

Iterpipes is "A library for running shell pipelines using shell-like syntax" and guess what? It's brilliant.

Here's an example from its PYPI page:

# Total lines in *.py files under /path/to/dir,
# use safe shell parameters formatting:

>>> total = cmd(
...     'find {} -name {} -print0 | xargs -0 wc -l | tail -1 | awk {}',
...     '/path/to/dir', '\*.py', '{print $1}')
>>> run(total | strip() | join | int)
315

Here's how that would look in shell:

find /path/to/dir -name '*.py' -print0 | xargs -0 wc -l | tail -1 | awk '{print $1}'

You may say the shell version looks better. That's an illusion caused by the evil that is shell scripting: the shell version is buggy.

Why is it buggy? Because if I control what's inside /path/to/dir I can make that neat little shell command fail [1], but at least in python I can handle errors!

Also, in most versions you could attempt to write, this command would be unsafe because quoting and escaping in shell is insane!

The iterpipes version uses the equivalent of SQL prepared statements which are much safer.

It's nearly impossible to do such a command in pure shell and be sure it's safe.

Also, the shell version produces a string instead of an integer, which sucks if you intend to do anything with it.

And the most important benefit is, of course, not when you try to make python act like a shell, but when you can stop pretending shell is a real programming language.

Consider this gem from Arch Linux's /etc/rc.shutdown script. Here, DAEMONS is a list of things that started on boot, and this script is trying to shut them down in reverse order, unless the daemon name starts with "!":

# Shutdown daemons in reverse order
let i=${#DAEMONS[@]}-1
while [ $i -ge 0 ]; do
        if [ "${DAEMONS[$i]:0:1}" != '!' ]; then
                ck_daemon ${DAEMONS[$i]#@} || stop_daemon ${DAEMONS[$i]#@}
        fi
        let i=i-1
done

Nice uh?

Now, how would that look in python (I may have inverted the meaning of ck_daemon)?

# Shutdown daemons in reverse order
for daemon in reversed(DAEMONS):
    if daemon[0]=='!':
        continue
    if ck_daemon(daemon):
        stop_daemon(daemon)

Where stop_daemon used to be this:

stop_daemon() {
    /etc/rc.d/$1 stop
}

And will now be this:

def stop_daemon(daemon):
    run(cmd('/etc/rc.d/{} stop',daemon))

So, come on, people, we are in the 21st century, and shell scripting sucked in the 20th already.

[1] I leave that as exercise for the reader.

Migrating from Haloscan to Disqus (if you can comment on it, it worked ;-)

Introduction

If you are a Haloscan user, and are starting to wonder what can you do... this page will explain you a way to take your comments to Disqus, another free comment service.

A few days ago, Haloscan announced they were stopping their free comment service for blogs. Guess what service has in it the comments of the last 9 years of this blog? Yes, Haloscan.

They offered a simple migration to their Echo platform, which you have to pay for. While Echo looks like a perfectly nice comment platform, I am not going to spend any money on this blog if I can help it, since it already eats a lot of my time.

Luckily, the guys at Haloscan allow exporting the comments (that used to be only for their premium accounts), so thanks Haloscan, it has been nice!

So, I started researching where I could run to. There seems to be two large free comment systems:

Keep in mind that my main interest lays in not losing almost ten years of comments, not on how great the service is. That being said, they both seem to offer roughly the same features.

Let's consider how you can import comments to each service:

  • Disqus: It can import from blogger and some other hosted blog service. Not from Haloscan.
  • Intense Debate: Can import from some hosted services, and from some files. Not from the file Haloscan gave me.

So, what is a guy to do? Write a python program, of course! Here's where Disqus won: they have a public API for posting comments.

So, all I have to do then is:

  1. Grok the Disqus API
  2. Grok the Haloscan comments file (it's XML)
  3. Create the necessary threads and whatever in Disqus
  4. Post the comments from Haloscan to Disqus
  5. Hack the blog so the links to Haloscan now work for Disqus

Piece of cake. It only took me half a day, which at my current rates is what 3 years of Echo would have costed me, but where's the fun in paying?

So, let's go step by step.

1. Grok the Disqus API

Luckily, there is a reasonable Disqus Python Client library and docs for the API so, this was not hard.

Just get the library and install it:

hg clone https://[email protected]/IanLewis/disqus-python-client/
cd disqus-python-client
python setup.py install

The API usage we need is really simple, so study the API docs for 15 minutes if you want. I got almost all the tips I needed from this pybloxsom import script

Basically:

  1. Get your API Key
  2. You login
  3. You get the right "forum" (you can use a disqus account for more than one blog)
  4. Post to the right thread

2. Grok the Haloscan comments file

Not only is it XML, it's pretty simple XML!

Here's a taste:

<?xml version="1.0" encoding="iso-8859-1" ?>
<comments>
    <thread id="BB546">
      <comment>
        <datetime>2007-04-07T10:21:54-05:00</datetime>
        <name>superstoned</name>
        <email>jos[email protected]</email>
        <uri></uri>
        <ip>86.92.111.236</ip>
        <text><![CDATA[that is one hell of a cool website ;-)]]></text>
      </comment>
      <comment>
        <datetime>2007-04-07T16:14:53-05:00</datetime>
        <name>Remi Villatel</name>
        <email>[email protected]</email>
        <uri></uri>
        <ip>77.216.206.65</ip>
        <text><![CDATA[Thank you for these rare minutes of sweetness in this rough world...]]></text>
      </comment>
    </thread>
</comments>

So, a comments tag that contains one or more thread tags, which contain one or more comment tags. Piece of cake to traverse using ElementTree!

There is an obvious match between comments and threads in Haloscan and Disqus. Good.

3. Create the necessary threads and whatever in Disqus

This is the tricky part, really, because it requires some things from your blog.

  • You must have a permalink for each post
  • Each permalink should be a separate page. You can't have permalinks with # in the URL
  • You need to know what haloscan id you used for each post's comments, and what the permalink for each post is.

For example, suppose you have a post at http://ralsina.me/weblog/posts/ADV0.html and it has a Haloscan comments link like this:

<a href="javascript:HaloScan('ADV0');" target="_self"> <script type="text/javascript">postCount('ADV0');</script></a>

You know where else that 'ADV0' appears? In Haloscan's XML file, of course! It's the "id" attribute of a thread.

Also, the title of this post is "Advogato post for 2000-01-17 17:19:57" (hey, it's my blog ;-)

Got that?

Then we want to create a thread in Disqus with that exact same data:

  • URL
  • Thread ID
  • Title

The bad news is... you need to gather this information for your entire blog and store it somewhere. If you are lucky, you may be able to get it from a database, as I did. If not... well, it's going to be a lot of work :-(

For the purpose of this explanation, I will assume you got that data nicely in a dictionary indexed by thread id:

{
  id1: (url, title),
  id2: (url, title)
}

4. Post the comments from Haloscan to Disqus

Here's the code. It's not really tested, because I had to do several attempts and fixes, but it should be close to ok (download).

#!/usr/bin/python
# -*- coding: utf-8 -*-

# Read all comments from a CAIF file, the XML haloscan exports

from disqus import DisqusService
from xml.etree import ElementTree
from datetime import datetime
import time


# Obviously these should be YOUR comment threads ;-)
threads={
    'ADV0': ('http://ralsina.me/weblog/posts/ADV0.html','My first post'),
    'ADV1': ('http://ralsina.me/weblog/posts/ADV1.html','My second post'),
    }

key='USE YOUR API KEY HERE'
ds=DisqusService()
ds.login(key)
forum=ds.get_forum_list()[0]

def importThread(node):
    t_id=node.attrib['id']

    # Your haloscan thread data
    thr_data=threads[t_id]

    # A Disqus thread: it will be created if needed
    thread=ds.thread_by_identifier(forum,t_id,t_id)['thread']

    # Set the disqus thread data to match your blog
    ds.update_thread(forum, thread, url=thr_data[0], title=thr_data[1])


    # Now post all the comments in this thread
    for node in node.findall('comment'):
        dt=datetime.strptime(node.find('datetime').text[:19],'%Y-%m-%dT%H:%M:%S')
        name=node.find('name').text or 'Anonymous'
        email=node.find('email').text or ''
        uri=node.find('uri').text or ''
        text=node.find('text').text or 'No text'

        print '-'*80
        print 'Name:', name
        print 'Email:', email
        print 'Date:', dt
        print 'URL:', uri
        print
        print 'Text:'
        print text

        print ds.create_post(forum, thread, text, name, email,
                                   created_at=dt, author_url=uri)
        time.sleep(1)

def importComments(fname):
    tree=ElementTree.parse(fname)
    for node in tree.findall('thread'):
        importThread(node)


# Replace comments.xml with the file you downloaded from Haloscan
importComments('comments.xml')

Now, if we are lucky, you already have a nice and fully functioning collection of comments in your Disqus account, and you should be calm knowing you have not lost your data. Ready for the final step?

Why I STILL use Arch Linux

Yesterday I had one of those moments where I feel very happy about my distro of choice, Arch Linux. Since the last time I posted about Arch seems to have been over two years ago (time flies when you are having fun!), I think it's time to explain it.

I wanted to test rst2pdf against reportlab from SVN, wordaxe from SVN and docutils from SVN, and I wanted it to be simple.

Solution: I just packaged them in AUR!

Now, whenever I need to check rst2pdf agains wordaxe trunk, I just need to yaourt -S python-wordaxe-svn and I can go back to stable wordaxe with yaourt -S python-wordaxe.

The svn package will always be the current trunk without any modifications, and I can switch back and forth in about 45 seconds, without messing up my system's packages.

Also, I can keep my installed SVN packages updated by doing yaourt -Su --devel every now and then.

How would I have done that using Debian or a RPM distro? I suppose by going around the packaging system (which I hate) or by doing a private repo (which is so ... lame?) or by doing a public repo (which is freaking work).

Really, if you are a coder, I can't think of a Linux distro that makes life easier than Arch. Pretty much everything is there (12K packages in unsupported!) and if it isn't, it's a 5-minute job to slap it into AUR and help the community.

Suppose you are doing a KDE app. On most distros you need to install your own from-source copy of kdelibs to have the latest and make sure it's not screwed by distro-specific patches.

On Arch? Patching upstream is frowned upon. Not having the latest version is frowned upon. So it's pretty much the ideal environment to develop against KDE, or GNOME, or PyQt or whatever.

If my life was not 150% committed already, I would try to become an Arch developer, or at least a TU (Trusted User). Maybe next life!

Outlook, IMAP and Exchange

Ok, I will post this just in case some poor soul needs to try doing it. You tell me if it makes any sense.

  • When installing Outlook 2000, you can choose support for "Groupware only", "Internet only" or "both".
  • Internet Only lets you create IMAP or POP3 accounts, but no Exchange accounts.
  • Groupware Only lets you use Exchange, but no IMAP or POP3.
  • Both gives you only Exchange and POP3, but no IMAP.

Here is how you can have Exchange and IMAP at the same time:

  1. Close Outlook.
  2. Go to control panel.
  3. There you have a "Mail" icon that was installed by Office.
  4. Use that icon to create the IMAP account.

And some people dare say windows is easy to manage :-P

Yes, I know, old software, whatever. It's not my fault if upgrading the freaking mail client costs so much money they won't do it becaue they have to upgrade the whole puddlejumping office suite.

PS: Thanks to the guy that told me how it's done ;-)

PS2: How to make Outlook show more than 100 results from LDAP: http://support.microsoft.com/kb/262848

A hard-to-block spammer: I need help.

Many of my clients have been spammed by La Capitana Real Estate lately. And I mean many. Hundreds.

However, they seem to have found a way to spam that works. And that sucks.

They have created a Google Group, added all their victims there, and let google do the dirty work.

What's the problem?

  1. Google group mails are not blockable at SMTP-level because their senders contain a sort of hash and the recipient address, and no group name. That's incredibly stupid in google's part.
  2. The messages they send are huge (6MB and up) so spamassassin can not process them. The SA docs say this will not happen because of "the economics of spam". Well, it happens when you make google do it!
  3. I don't want to go back to the old days of keeping a local queue-level address blacklist. That's awful!

I have complained to google, I have complained to the spammers, even by phone. They use the standard defense of "we are just inviting people". "They can unsubscribe if they want to". "This is not spam"

Noone does anything.

What's the next step? I can't blacklist google groups!

Why no packaging software should replace your config files

When you upgrade a piece of software on Linux, there are two paths it can go when there are incompatible changes in the config files (ok, 3 paths, Debian asks you what to do):

  1. The "rpmnew" way: install the new config file as "whatever.rpmnew", which means the softwarewill break immediately, but that's ok, because you are doing upgrades, so you should be watching already.
  2. The "rpmsave" way: replace the old file and save a copy as "whatever.rpmsave".

This has two problems:

  1. The software may fail or not, or fail in a subtle way, and you will not notice right away.

  2. Maybe the old file will be lost anyway:

    lrwxrwxrwx  1 root root 32 jul 15 22:41 /etc/named.conf -> /var/named/chroot/etc/named.conf
    lrwxrwxrwx  1 root root 32 jul 15 22:36 /etc/named.conf.rpmsave -> /var/named/chroot/etc/named.conf
    

In this case the "file" was a symlink, so by "saving a copy" it only saved another symlink to the soon-to-be-overwritten file.

And that's why, ladies and gentlemen, the rpmnew way is the good way.

Adding MSN notifications to Argus

I am a user of Argus as a monitoring software. Since it's very flexible and easy to extend, I wanted to add MSN alerts, the same way I had added SMS alerts a while ago. It was easier than I thought!

  1. Install msnlib

  2. Install the example msnbot, modified so do_work is like this:

    def do_work():
       """
       Here you do your stuff and send messages using m.sendmsg()
       This is the only place your code lives
       """
    
       # wait a bit for everything to settle down (sync taking efect
       # basically)
       time.sleep(15)
    
       msg=sys.stdin.read()
       for d in sys.argv[3].split('+'):
               print m.sendmsg(d,msg)
    
    
       # give time to send the messages
       time.sleep(30)
    
       # and then quit
       quit()
    
  3. Define a custom notification in argus like this:

    Method "msn" {
       command: /usr/local/bin/msnbot [email protected] mypass %R >/tmp/XXX 2>&1
       send: %M\n
    }
    
  4. Wherever you want MSN notifications, add this (on notify or escalate directives, using as many guys MSN addresses as you need):

    msn:[email protected][email protected]
    

That's it.

Wine kinda works nowadays

I had stopped trying to run stuff in WINE a few years ago,because pretty much nothing worked.

You know what? They areright about approaching 1.0, I had to try a few windowsy things lately, and each one has worked just fine.

Kudos wine people!

PS: can you make the widgets look less ugly, though? I heard rumours of a theming engine, but have no clue as to where one would look. Even something like GTK's clearlooks would be bearable.

Pop quiz!

Q: What happens if you have a SMTPA server, and you have a user called info, and then set its password to info?

A:

[[email protected] ~]# qmail-qstat
messages in queue: 151369
messages in queue but not yet preprocessed: 9

Lesson:

  • Yes, there is a reason why those passwords are not available by default.
  • I really should limit the per-account outgoing mail by default in all servers. Let's start working on it.

Linux as a windows crutch: Sending SMS

Suppose you want to send SMS messages from windows through a bluetooth connection to a phone.

I am sure you can make it work. On the other hand, I already had it working on Linux... so you can just use this on a friendly Linux box, and send SMS messages by accessing a special URL:

#!/usr/bin/env python
from colubrid import BaseApplication, HttpResponse, execute
import os

class SMSApplication(BaseApplication):

  def process_request(self):
      numero = self.request.args.get('numero')
      mensaje = self.request.args.get('mensaje')
      [entrada,salida]=os.popen4('/usr/bin/gnokii --sendsms %s'%numero,mode='rw')
      entrada.write(mensaje)
      entrada.flush()
      entrada.close()
      msg=salida.read()
      response = HttpResponse(msg)
      response['Content-Type'] = 'text/plain'
      return response

if __name__ == '__main__':
  execute(SMSApplication,debug=True, hostname='mybox.domain.internal', port=8080,reload=True)

If someone opens http://mybox.domain.internal:8080/?numero=1234?mensaje=hola%20mundo it sends "hola mundo" to the 1234 number.

I suppose I could call this a web telephony service or somesuch, but it's actually just the 5'solution that came to mind.

It uses a silly little not-a-web-framework called colubrid instead of something you may know, because I wanted to keep it simple, and it doesn't get much simpler than this.