Skip to main content

Ralsina.Me — Roberto Alsina's website

Posts about linux (old posts, page 18)

The day we saw the dinosaur (an Ada Lovelace Day story)

To­day, March 24th is Ada Lovelace day, a day of blog­ging to cel­e­brate the achieve­ments of wom­en in tech­nol­o­gy and sci­ence.. I am tak­ing the lib­er­ty to tag this as python so it ap­pears in the right plan­et­s, but that's just to pro­mote Ada Lovelace day. Sor­ry 'bout that.

I will write about the on­ly per­son who ev­er taught me pro­gram­ming, Clau­di­a. I was young, so the earth was still luke­war­m, the day we saw the di­nosaur.

I was just a green sopho­more in the School of Chem­i­cal En­gi­neer­ing where, para­dox­i­cal­ly I would nev­er take a chem­istry class, be­ing an ap­plied math stu­dent and all that, and at the time "per­son­al com­put­er­s" were a nov­el­ty, a toy of the up­per mid­dle class.

We had spent the first two months of the se­mes­ter learn­ing how to pro­gram the ob­vi­ous way: writ­ing as­sem­bler for a fic­tion­al ma­chine on pa­per by hand, when Clau­dia broke the news, we were go­ing to see a re­al com­put­er.

No, not a PC, not even an XT, but a re­al com­put­er, the one re­al com­put­er in all the uni­ver­si­ty, and you could hear the type switch­ing to bold as she spoke about it. Sad­ly it was not as re­al as the one at the re­search fa­cil­i­ty (A Mini­VAX!) but it was a re­al enough PDP.

We would not be al­lowed to ac­tu­al­ly use it un­til the fol­low­ing year, but ... well, it was still some­thing spe­cial.

I had been programming for years, even for a year before I saw my first (seriosuly not real) computer, I had followed BASIC programs in my head for days, imagining the space invaders float on the screen of my mind, and stepped into writing machine code inside REM statements in my Timex Sinclair 1000 onto the luxury of a C64, but never noone had taught me anything.

Our small class (maybe 10 stu­dents) spent end­less hours do­ing things like tra­verse a ma­trix, first by rows, thn by column­s, then in a spi­ral from the top-left, writ­ing pro­grams that fol­lowed our end­less source of al­go­rithm­s, the nu­mer­i­cal so­lu­tions guide.

First as­sem­bler, then For­tran, we learned.

She was my Mr. Miyag­i, I was a het­ero­sex­u­al Ralph Mac­chio, and I fig­ured out the most im­por­tant thing about pro­gram­ming: I was aw­ful at it.

Over the next 20 years that sit­u­a­tion has been slow­ly im­prov­ing, but I nev­er again had some­one teach me pro­gram­ming. Clau­dia had al­ready taught me ev­ery­thing I need­ed to know, that code can al­ways im­prove, that there's more than one way to skin a cat.

That the di­nosaur was re­al and that some day soon my com­put­er would be faster and nicer than the di­nosaur was then, and that pro­gram­ming was cool, and that if I could find a way to draw a poly­no­mi­al graph hor­i­zon­tal­ly on a print­er with­out ev­er hav­ing the whole graph in mem­o­ry (it did­n't fit), those fu­ture com­put­ers would do awe­some things, and that I was one of the many who would help bring that to re­al­i­ty.

That talk­ing about code was fun in it­self, that you could make a mod­est liv­ing and be hap­py about it, that you could in any case make jig­saw puz­zles in your spare time and keep on teach­ing or what­ev­er.

And lat­er the di­nosaur's bones were scav­enged in­to a line of racks hold­ing router­s, and its glass ter­mi­nals are de­stroyed, and the gold in its teeth was stolen and the rare bus ca­bles sol­d, and its cir­cuits scrapped, but I saw the di­nosaur alive, and Clau­dia taught me how to make it jump, and for that, I will al­ways be grate­ful.

Marave 0.5 is out!

Just up­load­ed Mar­ave 0.5 to the usu­al place. Mar­ave is a re­laxed, fullscreen text ed­i­tor that tries not to dis­tract you.

It even in­cludes a sim­ple mu­sic play­er just so you don't have to think about switch­ing to an­oth­er ap­pli­ca­tion!

This re­lease has sev­er­al bugs fixed, and looks a bit nicer.

The main new fea­ture is ... in­ter­na­tion­al­iza­cion. It now in­cludes a span­ish trans­la­tion on­ly, but if you want to help trans­lat­ing it to any oth­er lan­guage, please step for­ward!

There are on­ly 60 phras­es to trans­late, so it should­n't be more than one hour of work.

Here's a screen­shot of this re­lease:

marave7

Mar­ave is free soft­ware un­der the GPLv2, and should work on any plat­form where PyQt work­s, which means Win­dows, Mac, and Unix-­like op­er­at­ing sys­tem­s, at least.

Packaging and shipping is HARD

I have worked re­al­ly hard on Mar­ave, a full screen ed­i­tor in the style of ommwriter, Dark­Room, Write­Room, py­Room, etc. I have worked very hard and I want users to use it.

Or not even that, I want them to have a chance of us­ing it.

That means I want it to work on Win­dows (and maybe OSX some day, too, if some­one helps me). Which mean­s, I have to pak­age it for win­dows.

Let's do a quick com­par­i­son here from the points of view of the us­er and the de­vel­op­er.

The User, In Linux

This is in Arch Lin­ux, which is what I use, in oth­er Lin­ux vari­ants it will be pret­ty much the same once Mar­ave is a bit more well known.

yaourt -S marave-svn --noconfirm

That gets the code from SVN (right now it's the best way, lat­er I will pack­age re­leas­es, too), all re­quired de­pen­den­cies, builds and in­stall­s. It takes all of 15 sec­onds in my note­book.

Af­ter that, you have a ful­ly work­ing Mar­ave.

In case it's not pack­aged for your dis­tro, just in­stall PyQt (which sure­ly is) and run this com­mand:

easy_install marave

The User, in Windows

You go to http://­mar­ave.­google­code.­com, click on "Mar­ave-0.5.win32.ex­e" (Not linked yet, it's not fin­ished), then down­load a 10MB pro­gram. That is a 10MB pro­gram be­cause win­dows does­n't be­lieve in pack­ages and de­pen­den­cies. On Lin­ux, a Mar­ave pack­age could be un­der 1MB (most of it im­ages), and not be ex­e­cutable, just da­ta.

Of course nowa­days web browsers don't ac­tu­al­ly run pro­grams on down­load, so... let's see it as a gallery!

110111105613-My-Desktop

Yes, save it.

11011111220-My-Desktop

Dou­ble click to open it

11011111417-My-Desktop

Yes, I agree

11011111514-My-Desktop

Sure, what­ev­er

1101111167-My-Desktop

Nice...

11011111750-My-Desktop

Good to hear!

Now, this Mar­ave that just got in­stalled may or may not cur­rent­ly work be­cause of a miss­ing MSVCR90.DLL but that's for the next sec­tion...

The Developer, in Linux

First, here's the big­gest prob­lem a Lin­ux pack­ager can have:

Since Mar­ave is a new ap­p, and I de­vel­op it in the rather cut­ting-edge Arch Lin­ux, it us­es some newish fea­tures on­ly avail­able in re­cent ver­sions of Qt. In fac­t, it does­n't work with PyQt < 4.6, which is not avail­able in some slow dis­tros, like De­bian, or even in a not-lat­est Ubun­tu.

So­lu­tion? Well, I could just ig­nore it, but what the heck, let's fix it in­stead!

Thanks to PyIn­staller it's not even hard to do, here's the spec file:

a = Analysis([os.path.join(HOMEPATH,'support/_mountzlib.py'), os.path.join(HOMEPATH,'support/useUnicode.py'), 'marave/main.py'],
            pathex=['/home/ralsina/trunk/trunk'])

pyz = PYZ(a.pure)
exe = EXE(pyz,
        a.scripts,
        exclude_binaries=1,
        name=os.path.join('build/pyi.linux2/main', 'marave.exe'),
        debug=False,
        strip=False,
        upx=True,
        console=0 )

coll = COLLECT( exe,
            a.binaries,
            [('radios.txt','marave/radios.txt','DATA')],
            Tree('marave/icons','icons'),
            Tree('marave/backgrounds','backgrounds'),
            Tree('marave/clicks','clicks'),
            Tree('marave/stylesheets','stylesheets'),
            Tree('marave/themes','themes'),
            a.zipfiles,
            a.datas,
            strip=False,
            upx=True,
            name=os.path.join('dist', 'marave'))

Use this, and PyIn­staller will pro­duce a nice fold­er full of ev­ery­thing Mar­ave needs to run on any Lin­ux.

OTO­H, if you can re­ly on a re­cent PyQt be­ing avail­able, it's al­so sim­ple. Here's a pack­ag­ing con­fig­u­ra­tion for a sim­i­lar pack­age in Arch Lin­ux (I must con­fess not hav­ing done one for Mar­ave yet). For oth­er dis­tri­bu­tions it should be about as sim­ple, if more ver­bose, and some­one else prob­a­bly does it for you:

# Contributor: Roberto Alsina <ralsina@kde.org>
pkgname=python-rst2pdf
pkgver=0.12.1
pkgrel=4
pkgdesc="Create PDFs from simple text markup, no LaTeX required."
arch=('i686' 'x86_64')
url="http://rst2pdf.googlecode.com"
license=('custom')
depends=('python' 'setuptools' 'docutils' 'pygments' 'python-reportlab' 'python-simplejson' 'pil')
source=(http://rst2pdf.googlecode.com/files/rst2pdf-$pkgver.tar.gz LICENSE.txt)
optdepends=('uniconvertor: vector images support'
            'python-svglib: SVG support'
            'python-wordaxe: hyphenation'
            'pythonmagick: PDF images support')
build() {
cd $startdir/src/rst2pdf-$pkgver
python setup.py install --root=$startdir/pkg || return 1
install -D ../LICENSE.txt $startdir/pkg/usr/share/licenses/python-rst2pdf/COPYING
install -D doc/rst2pdf.1 $startdir/pkg/usr/share/man/man1/rst2pdf.1
}
md5sums=('ea6beda9a46f34ba42c4c94d48cc607a'
        '416f8046c66b9476cdbacda69a673afe')

And that's all you need to know about the process of pack­ag­ing your app for Lin­ux. It's easy to do, and most of the time, easy to do right!

Now, let's go to our fi­nal sec­tion...

Windows for the developer

First, re­mem­ber that of re­ly­ing on the sys­tem's ver­sion of Qt? For­get it, there is no sys­tem ver­sion avail­able. And no python ei­ther. And noone is go­ing to in­stall it or your ap­p, so it's "ship ev­ery­thing your­self" mod­e, or noth­ing.

But any­way, PyIn­staller works for Win­dows too! So, us­ing the same spec file, it work­s. Right?

Well, no beause of two prob­lem­s.

Problem 1: You need an installer

Users are not go­ing to open a zip some­where, then do a short­cut to the bi­na­ry on Win­dows, so you need to do some op­er­a­tions, and that means an in­stall­er.

Here's what I came up with to use NSIS, a free in­stall­er cre­ator for Win­dows:

;--------------------------------
;Include Modern UI

!include "MUI2.nsh"

;--------------------------------
;General

;Name and file
Name "Marave"
OutFile "Marave-0.5.win32.exe"

;Default installation folder
InstallDir "$LOCALAPPDATA\Marave"

;Get installation folder from registry if available
InstallDirRegKey HKCU "Software\Marave" ""

;Request application privileges for Windows Vista
RequestExecutionLevel user

;--------------------------------
;Interface Settings

!define MUI_ABORTWARNING

;--------------------------------
;Pages

!insertmacro MUI_PAGE_LICENSE "LICENSE"
!insertmacro MUI_PAGE_DIRECTORY
!insertmacro MUI_PAGE_INSTFILES

!insertmacro MUI_UNPAGE_CONFIRM
!insertmacro MUI_UNPAGE_INSTFILES

;--------------------------------
;Languages

!insertmacro MUI_LANGUAGE "English"

;--------------------------------
;Installer Sections

Section "Install"

SetOutPath "$INSTDIR"
File /r "dist\marave"


;Store installation folder
WriteRegStr HKCU "Software\Marave" "" $INSTDIR

;Create uninstaller
WriteUninstaller "$INSTDIR\Uninstall.exe"

;Create shortcuts
CreateDirectory $SMPROGRAMS\Marave
CreateShortCut "$SMPROGRAMS\Marave\Marave.lnk" "$INSTDIR\marave\marave.exe" ; use defaults for parameters, icon, etc.
CreateShortCut "$SMPROGRAMS\Marave\Uninstall Marave.lnk" "$INSTDIR\Uninstall.exe" ; use defaults for parameters, icon, etc.

SectionEnd


;--------------------------------
;Uninstaller Section

Section "Uninstall"

Delete "$INSTDIR\Uninstall.exe"
RMDir /r "$INSTDIR"

DeleteRegKey /ifempty HKCU "Software\Marave"

SectionEnd

It's com­pa­ra­ble to the ef­fort of build­ing a pack­ag­ing file, re­al­ly, ex­cept ev­ery time you want to test it... you in­stall it. There is no way (AFAIC­S) to see what's in­side the in­stall­er ex­cept run­ning it!

When things fail, you get no er­ror mes­sages, at least not the kind that is use­ful for a de­vel­op­er, the guy that needs to know what went wrong.

And af­ter it's fin­ished, you may end with a non-­work­ing pro­gram be­cause of...

Problem 2: system libraries don't exist

Python 2.6 bi­na­ries are built us­ing Vis­ual Stu­dio. That means they re­quire the Vis­ual Stu­dio Run­time, specif­i­cal­ly MSVCR90.DL­L. That con­tains what on Lin­ux would be con­sid­ered part of libc. (lin­ux guy: imag­ine apps that de­pend on a spe­cif­ic libc... hard to do!)

On Lin­ux that's part of the sys­tem. Fur­ther, if you want­ed, you can re­dis­tribute it. On Win­dows... well, it's a bit dif­fer­en­t.

  1. It's part of the "Vi­­su­al C++ re­dis­­tribu­ta­bles"

  2. In­­stalling that does­n't guar­an­­tee it will work (yes, I have tried)

  3. The li­­cense for those 're­dis­­tribu­ta­bles' says you can't make them avail­able for down­load.

    I have been told that in­­­clud­ing that in your in­­stal­l­er is fine and dandy, but how is that not mak­ing them avail­able for down­load?

So what can you do when you need a li­brary and can't ship it and the us­er won't in­stall it?

Well, that's why there is no Win­dows bi­na­ry of Mar­ave yet. Of course if any­one can help, I'd be re­al­ly, re­al­ly hap­py!

Happy 10th blogiversary to me!

Since yes­ter­day this blog is ten years old so, time for some his­to­ry.

It all start­ed in ad­voga­to where you could still read it to­day! (Please read it here in­stead ;-)

Then it moved to PyDS an ear­ly python desk­top blog plat­form with a web in­ter­face, and was host­ed in PyC­S, a free ser­vice.

Then PyCS kin­da died, and I start­ed gen­er­at­ing a stat­ic blog and host­ing it in my IS­P's free host­ing. That sucked bad.

Then I start­ed my own com­pa­ny, and I had my own server­s, so I start­ed host­ing it there (even to­day this blog is com­plete­ly stat­ic HTM­L!)

Then PyDS start­ed act­ing weird, so I wrote my own blog­ging soft­ware, which is a re­al mess, per­haps 25% fin­ished, but it does things ex­act­ly the way I like them.

Cur­rent­ly, this blog is syn­di­cat­ed in Plan­e­ta PyAr, Plan­et Python, Plan­et Qt, Plan­e­ta LUGLI, and a cou­ple oth­er places.

This year, I de­cid­ed to make the blog com­plete­ly bilin­gual (English and Span­ish), but I hate trans­lat­ing it.

Ac­cord­ing to the stats I have avail­able, the blog is in av­er­age more pop­u­lar now than ev­er (but yes, my most pop­u­lar posts were years ago ;-)

stats

These are the most pop­u­lar pages in the last year:

Lesson­s:

  1. I need to write more about Qt and/or start flame­wars with clue­­less IT writ­ers

  2. I need to search for an­­cient ma­te­ri­al and de­p­re­­cate it

  3. Hav­ing your own host­ing and blog­ging soft­­ware is neat

  4. 10 years is a lot of time: 860 posts (or 913, de­pend­ing on how you coun­t)

With iterpipes, python is ready to replace bash for scripting. Really.

This has been a pet peeve of mine for years: pro­gram­ming shell scripts suck. They are ug­ly and er­ror prone. The on­ly rea­son why we still do it? There is no re­al re­place­men­t.

Or at least that was the case, un­til to­day I met iter­pipes at python.red­dit.­com

Iter­pipes is "A li­brary for run­ning shell pipe­lines us­ing shel­l-­like syn­tax" and guess what? It's bril­liant.

Here's an ex­am­ple from its PYPI page:

# Total lines in *.py files under /path/to/dir,
# use safe shell parameters formatting:

>>> total = cmd(
...     'find {} -name {} -print0 | xargs -0 wc -l | tail -1 | awk {}',
...     '/path/to/dir', '\*.py', '{print $1}')
>>> run(total | strip() | join | int)
315

Here's how that would look in shel­l:

find /path/to/dir -name '*.py' -print0 | xargs -0 wc -l | tail -1 | awk '{print $1}'

You may say the shell ver­sion looks bet­ter. That's an il­lu­sion caused by the evil that is shell script­ing: the shell ver­sion is bug­gy.

Why is it buggy? Because if I control what's inside /path/to/dir I can make that neat little shell command fail [1], but at least in python I can handle errors!

Al­so, in most ver­sions you could at­tempt to write, this com­mand would be un­safe be­cause quot­ing and es­cap­ing in shell is in­sane!

The iter­pipes ver­sion us­es the equiv­a­lent of SQL pre­pared state­ments which are much safer.

It's near­ly im­pos­si­ble to do such a com­mand in pure shell and be sure it's safe.

Al­so, the shell ver­sion pro­duces a string in­stead of an in­te­ger, which sucks if you in­tend to do any­thing with it.

And the most im­por­tant ben­e­fit is, of course, not when you try to make python act like a shel­l, but when you can stop pre­tend­ing shell is a re­al pro­gram­ming lan­guage.

Consider this gem from Arch Linux's /etc/rc.shutdown script. Here, DAEMONS is a list of things that started on boot, and this script is trying to shut them down in reverse order, unless the daemon name starts with "!":

# Shutdown daemons in reverse order
let i=${#DAEMONS[@]}-1
while [ $i -ge 0 ]; do
        if [ "${DAEMONS[$i]:0:1}" != '!' ]; then
                ck_daemon ${DAEMONS[$i]#@} || stop_daemon ${DAEMONS[$i]#@}
        fi
        let i=i-1
done

Nice uh?

Now, how would that look in python (I may have in­vert­ed the mean­ing of ck­_­dae­mon)?

# Shutdown daemons in reverse order
for daemon in reversed(DAEMONS):
    if daemon[0]=='!':
        continue
    if ck_daemon(daemon):
        stop_daemon(daemon)

Where stop_­dae­mon used to be this:

stop_daemon() {
    /etc/rc.d/$1 stop
}

And will now be this:

def stop_daemon(daemon):
    run(cmd('/etc/rc.d/{} stop',daemon))

So, come on, peo­ple, we are in the 21st cen­tu­ry, and shell script­ing sucked in the 20th al­ready.


Contents © 2000-2023 Roberto Alsina