Skip to main content

Ralsina.Me — Roberto Alsina's website

With iterpipes, python is ready to replace bash for scripting. Really.

This has been a pet peeve of mine for years: pro­gram­ming shell scripts suck. They are ug­ly and er­ror prone. The on­ly rea­son why we still do it? There is no re­al re­place­men­t.

Or at least that was the case, un­til to­day I met iter­pipes at python.red­dit.­com

Iter­pipes is "A li­brary for run­ning shell pipe­lines us­ing shel­l-­like syn­tax" and guess what? It's bril­liant.

Here's an ex­am­ple from its PYPI page:

# Total lines in *.py files under /path/to/dir,
# use safe shell parameters formatting:

>>> total = cmd(
...     'find {} -name {} -print0 | xargs -0 wc -l | tail -1 | awk {}',
...     '/path/to/dir', '\*.py', '{print $1}')
>>> run(total | strip() | join | int)
315

Here's how that would look in shel­l:

find /path/to/dir -name '*.py' -print0 | xargs -0 wc -l | tail -1 | awk '{print $1}'

You may say the shell ver­sion looks bet­ter. That's an il­lu­sion caused by the evil that is shell script­ing: the shell ver­sion is bug­gy.

Why is it buggy? Because if I control what's inside /path/to/dir I can make that neat little shell command fail 1, but at least in python I can handle errors!

Al­so, in most ver­sions you could at­tempt to write, this com­mand would be un­safe be­cause quot­ing and es­cap­ing in shell is in­sane!

The iter­pipes ver­sion us­es the equiv­a­lent of SQL pre­pared state­ments which are much safer.

It's near­ly im­pos­si­ble to do such a com­mand in pure shell and be sure it's safe.

Al­so, the shell ver­sion pro­duces a string in­stead of an in­te­ger, which sucks if you in­tend to do any­thing with it.

And the most im­por­tant ben­e­fit is, of course, not when you try to make python act like a shel­l, but when you can stop pre­tend­ing shell is a re­al pro­gram­ming lan­guage.

Consider this gem from Arch Linux's /etc/rc.shutdown script. Here, DAEMONS is a list of things that started on boot, and this script is trying to shut them down in reverse order, unless the daemon name starts with "!":

# Shutdown daemons in reverse order
let i=${#DAEMONS[@]}-1
while [ $i -ge 0 ]; do
        if [ "${DAEMONS[$i]:0:1}" != '!' ]; then
                ck_daemon ${DAEMONS[$i]#@} || stop_daemon ${DAEMONS[$i]#@}
        fi
        let i=i-1
done

Nice uh?

Now, how would that look in python (I may have in­vert­ed the mean­ing of ck­_­dae­mon)?

# Shutdown daemons in reverse order
for daemon in reversed(DAEMONS):
    if daemon[0]=='!':
        continue
    if ck_daemon(daemon):
        stop_daemon(daemon)

Where stop_­dae­mon used to be this:

stop_daemon() {
    /etc/rc.d/$1 stop
}

And will now be this:

def stop_daemon(daemon):
    run(cmd('/etc/rc.d/{} stop',daemon))

So, come on, peo­ple, we are in the 21st cen­tu­ry, and shell script­ing sucked in the 20th al­ready.

1

I leave that as ex­er­cise for the read­er.

tzulberti / 2009-12-23 20:21:

definitivamente te la pasas revisando pipy.....

Roberto Alsina / 2009-12-23 20:30:

No, estoy subscripto a python.reddit.com ... mucho menos laburo! ;-)

Dimitris Leventeas / 2009-12-23 22:22:

Excellent post!
Bash is obsolete. Long live Python!

elpargo / 2009-12-24 08:02:

Awesome thank you!

elpargo / 2009-12-24 08:02:

Just today I was ranting about bash and how it sucks so much!

Eric O LEBIGOT (EOL) / 2009-12-24 09:43:

Thank you for sharing.

IPython is also great for on-the-fly system administration:

>>> files = !ls *.py
etc.

Andrey Vlasovskikh / 2009-12-25 08:26:

Thanks for your review :) The library is still in active development and definitely requires more documentation, testing, cleaning up its API, etc.

Your example with Arch Linux init scripts is quite interesting, because scripting in Arch is one of my use cases for the library.

BTW, in run(cmd('/etc/rc.d/{} stop',daemon)), you should use call or check_call instead of run as run returns Iterable -> Iterable and here you just need to force immediate execution of a command. As I've already said, more docs are needed.

Roberto Alsina / 2010-01-14 12:58:

Thanks for the correction. That's what happens when I don't test the code I post!

Kenny Meyer / 2009-12-27 14:15:

I'm a Python lover and this post is a WOW! experience. Python scripts look **so much** better than Bash scripts do!

Also the *error handling* is a plus point of why to replace a dynamic programming language, like Python, with an old-school scripting language like Bash.

Maybe this was a tedious task when using the `sys` and `os` or `subprocess` module for executing commands and let's not talk about pipelining... that was very ugly. But with `iterpipes` this is a complete new experience.

Thanks for sharing!

Roberto Alsina / 2010-01-14 12:57:

There is a nice, long discussion of this post here (in czech): http://is.gd/6fstv

Google translation: http://is.gd/6a1bC

Joan / 2010-02-14 10:18:

Hi! You could be interested on Scripy. It has been built to replace to bash scripts, it's integrated with logging (all shell commands can be logged), to lets edit files (more easy than in bash), and more...

http://pypi.python.org/pypi...
http://bitbucket.org/ares/s...

Roberto Alsina / 2010-02-14 12:22:

Very interesting!

Chris2048 / 2011-04-13 02:07:

In your first example you use pipes in an infix notation; I can't find that iterpipes supports this, was it dropped?

phone number lookup / 2011-12-03 22:30:

this is really interesting viewpoint on the subject i might add

employment background check / 2011-12-27 23:30:

Man ... Beautiful . Amazing ... I will bookmark your website and use the your RSS feed also

cell phone lookup / 2012-01-17 05:52:


Your blog has the same post as another author but i like your better


Contents © 2000-2020 Roberto Alsina