Skip to main content

Ralsina.Me — Roberto Alsina's website

The lost opportunity in test coverage

Dis­claimer: This is a bit of a rant, but it's a friend­ly rant :-)

When peo­ple look at code cov­er­age, they are read­ing it wrong.

Suppose you have a class, something stupid, like your own implementation of a stack, called Stack. Because you are not a total monster, you have tests in your code right? In fact, you are claiming that you are doing TDD (Test Driven Development), or at least you like TDD, or you would like the idea of TDD, or, let's be honest here, you just say you are doing TDD, but what you do is you sprinkle the tests you feel are needed, which is largely OK, I am not going to judge you, you freak.

And then you add test cov­er­age check­s, and it says: 80%

What most peo­ple feel when they see that is dread. They see that 80% and feel "OM­FG, my tests suck! I don't have enough! If even 100% cov­er­age is not enough then this 80% means my code is an un­sta­ble piece of garbage!"

Well, no.

Whether your code is good or not is in­de­pen­dent of test­s. Tests give you the abil­i­ty to know if your code is crap or not... some­times. What tests re­al­ly give you (if they are not to­tal garbage in them­selves) is the con­fi­dence that you can change your code with­out sig­nif­i­cant­ly af­fect­ing the be­hav­iours the tests are test­ing.

So, if your tests of Stack ensure that:

  • Stack.push puts the element at the top
  • Stack.pop gets the top element
  • Your stack can hold as many el­e­ments as your re­quire­ment de­fines (may be in­finite)

Then what you implemented is a stack. Period. It works. It's fine. It may be inefficient, it may be ugly, who knows, but tests are not going to give you good taste. All they are going to do is ensure that Stack is, indeed, a stack, and behaves like a stack, and that when you stick your mittens in it and change things inside it it stays a stack.

Yet, your cov­er­age is 80%.

Should you add more test­s?

No.

You should delete 20% of your code.

Since code is a li­a­bil­i­ty and the as­set is the code's be­haviour, then that's what the first D in TDD is for.

Test Driv­en De­vel­op­men­t.

Use the tests to de­fine the be­hav­iour you wan­t. Then add code to im­ple­ment that be­hav­iour.

Don't chase use­less stats like cov­er­age.

If cov­er­age is not 100%, con­sid­er your test­s.

Is there be­hav­iour you want that is not rep­re­sent­ed as a sce­nario in a test?

If yes: then add test­s.

If not: re­move code.

And us­ing "cov­er­age is low" as an op­por­tu­ni­ty to delete code in­stead of adding tests is some­thing a lot of de­vel­op­ers mis­s.

CobraPy: a group of minimum viable things

A group of crows is called a mur­der of crows. A group of hares is called a coun­cil of hares.

In fac­t, that's just a bunch of things vic­to­ri­ans made up be­cause they had lots of free time and they had­n't in­vent­ed the In­ter­net yet, and most of those were nev­er ac­tu­al­ly wide­ly used.

BUT what's the name for a group of things that are in a min­i­mal­ly vi­able state?

Well, Co­braPy, my 80s-style python pro­gram­ming en­vi­ron­ment is slow­ly crawl­ing in­to be­com­ing one of those.

Of the com­po­nents I wan­t, I have one of each. They all suck but they suck in the same way a 3 year old play­ing pool suck­s. He will not be great but it's still cool.

  • I am go­ing to punt in hav­ing the ed­i­tor I want be­cause I can make do with Mi­cro for the time be­ing (works flaw­less­ly in my ter­mi­nal!)
  • The RE­PL is not great but it can do what it can do
  • The graph­ics serv­er works (although its API is lim­it­ed to draw­ing cir­cles)
  • The ter­mi­nal is bet­ter than ex­pect­ed, could re­al­ly be used as a dai­ly driv­er ex­cept for some pro­grams re­al­ly not lik­ing it.

And al­so, I have com­bined all the things so that you can start a win­dow that:

  • Is a ter­mi­nal
  • That runs the re­pl
  • Where you can use the graph­ics API
  • And it dis­plays in the same win­dow

What nex­t? I could think about what nex­t. Or ...

I could try to write a sim­ple game and im­ple­ment all the things that don't ex­ist.

Ex­cept for in­put. I need to solve how to do in­put. You see, the user-cre­at­ed pro­grams don't run in the same space as the win­dow. That's why we have a graph­ics pro­to­col. The pro­gram puts things in it, the win­dow reads them and graph­ics ap­pear.

But in­put needs to go the oth­er way around. So I need to add a sec­ond pro­to­col to send back events and it needs to be pret­ty fast. I don't think it's go­ing to be a prob­lem (us­er ac­tions hap­pen on­ly once ev­ery few dozens of mil­lisec­ond­s!) but af­ter that's done?

It's go­ing to be time for ...

Or ac­tu­al­ly, to fail at im­ple­ment­ing it, but im­prov­ing the plat­form in the process. Be­cause fail­ure is what im­prove­ments are made of.

CobraPy: bits and pieces

As it hap­pens in ear­ly stages in fun prod­uct­s, progress in Co­braPy has been both faster and slow­er than ex­pect­ed.

In the past few days a num­ber of things hap­pened:

I made that terminal a whole lot nicer

I al­ready had a ter­mi­nal but I fixed a num­ber of things.

  • The key­board han­dling is much bet­ter, it now rec­og­nizes pret­ty much all keys, which is al­ways a nice thing.
  • It sor­ta sup­ports things like "á" via mod­e-switch (no dead­keys sup­port prob­a­bly ev­er)
  • It has 24-bit col­or sup­port! I did­n't know ter­mi­nals could sup­port that kind of thing!

I added a graphics protocol to it!

Ter­mi­nals with graph­ics sup­port have a very long tra­di­tion. This is a VT55, re­leased in 1977 dis­play­ing graph­ic­s:

How did it work? Well you can read the pro­gram­mer's man­u­al if you wan­t, but ba­si­cal­ly you sent a con­trol se­quence that put it in "graph­ics mod­e" and then sent com­mands de­scrib­ing what to dis­play.

Sim­i­lar ideas with dif­fer­ent pro­to­col de­tails were used in many dif­fer­ent fu­ture ter­mi­nal­s, in­clud­ing ReG­IS graph­ics and Tek­tron­ix vec­tor graph­ics and you could even trace this all the way to a cur­rent Lin­ux desk­top's X11 graph­ic­s.

So, what did I do? Not that, ex­act­ly. I am cre­at­ing a side-chan­nel as a sort-of-R­PC where you send se­ri­al­ized python method names and ar­gu­ments.

I wrote a Python REPL

I want­ed an in­ter­ac­tive mode that was slight­ly friend­lier than Python comes with, but not some­thing over­whelm­ing and pow­er­ful like IPython or BPython.

I did some re­search, and found pt­python which is pret­ty awe­some, but still a bit too much awe­some.

And then I start­ed on a much, much lamer ver­sion of it. Still em­bri­on­ic, but it does work. I have some plans for it.

I learned a lot more about Raylib

All the graph­ics and ba­si­cal­ly ev­ery­thing you see in this project is done us­ing the awe­some raylib and a home­grown CF­FI bind­ing for it. I was not us­ing it right, now I use it bet­ter, and things that took sev­er­al hun­dredth sec­onds now take a few dozen mi­crosec­ond­s.

I integrated the whole thing, sorta

So, I in­te­grat­ed it enough that you can start the ter­mi­nal, launch the RE­PL in it, and use the graph­ics pro­to­col to draw some­thing!

What next

Now comes a round of in­te­gra­tion, cleanup and op­ti­miza­tion.

  • Work­ing code needs to be re­or­ga­nized
  • The ter­mi­nal us­es 66% of a CPU core, which is not ac­cept­able, but there's tons of low hang­ing fruit there.
  • Graph­ics pro­to­col needs to be able to do more things so it's in­ter­est­ing

Af­ter that will come a new round of fea­ture work, and so on for the next ... 10 years? If it goes well?

Here, have a terminal

Co­braPy is in large part about rein­vent­ing wheel­s. We do, af­ter al­l, have per­fect­ly fine 80s-style de­vel­op­ment en­vi­ron­ments in the mil­lions of 80s com­put­ers float­ing around, as well as em­u­la­tors, things like the Maximite and so on.

How­ev­er, I on­ly want to rein­vent fun wheel­s, so I am not do­ing a text ed­i­tor. And prob­a­bly gonna hack a ter­mi­nal-based RE­PL. And there's no way I am do­ing my own wifi con­fig tool (again) so, I should have a reg­u­lar, or­di­nary ter­mi­nal.

But the things I want that ter­mi­nal to do are ... un­usu­al, like raster graph­ic­s, and sprites. So ... why not do my own ter­mi­nal for this?

It's not the first ter­mi­nal I write! I wrote a dumb ter­mi­nal em­u­la­tor around 1998 and I wrote a very ba­sic one us­ing python, pygame and pyte a few months ago. It's on video!

But this time I want­ed a bet­ter one, since I want this to look good. So I went and wrote one.

It us­es many 3rd par­ty things, be­cause life is short.

  • Pyte for all the nit­ty grit­ty of ter­mi­nal em­u­la­tion
  • raylib for all the graph­ics
  • mon­oid is the font
  • This page helped me with things like "what was the es­cape code for left­????"

And it looks this nice:

Yes, NICE, Stu­ar­t.

This is why you don't run random stuff.

Yes­ter­day I was try­ing to fig­ure ou­ut some ob­scure in­put things in Co­braPy such as "de­tect­ing the En­ter key" and ran in­to a love­ly pack­age in PyP­I: Pym­put

What does it do? It lets you in­ject and read in­put events.

And by in­put events I mean mouse and­key­board.

And it does so in your whole ses­sion.

And you would­n't no­tice if it was do­ing that.

And it does this in 20 lines of friend­ly python code. Here, have a key­board snif­fer:

from pynput import keyboard

def on_press(key):
    try:
        print('alphanumeric key {0} pressed'.format(
            key.char))
    except AttributeError:
        print('special key {0} pressed'.format(
            key))

def on_release(key):
    print('{0} released'.format(
        key))
    if key == keyboard.Key.esc:
        # Stop listener
        return False

# Collect events until released
with keyboard.Listener(
        on_press=on_press,
        on_release=on_release) as listener:
    listener.join()

This is one of the rea­sons why Way­land (or Mir, I re­mem­ber Mir!) needs to hap­pen. It's triv­ial for any desk­top app to mon­i­tor ev­ery­thing you do. Of course nowa­days you al­so will see soft­ware ad­ver­tis­ing this, as a "fea­ture" where it's used to "mon­i­tor em­ploy­ee pro­duc­tiv­i­ty".

Be­cause re­mem­ber, of­ten things are on­ly il­le­gal when in­di­vid­u­als do them, if you are a com­pa­ny and are charg­ing for it, then Bob's your un­cle.


Contents © 2000-2025 Roberto Alsina