There is one thing worse than not having a test suite
UPDATE: There is *another* *better* test suite It is in YAML, though, so I need to parse it before I can use it, but that's my problem.
It's having a test suite that makes no sense.
I have written, for my ra-plugins project (you don't have to know what it is for this post anyway) a piece of code that tries to check mail senders using SPF.
SPF is an open standard. It has standard implementations. It has a test suite (http://www.schlitt.net/spf/tests/).
- The test suite says this:
spfquery -ip=192.0.2.1 -sender=05.spf1-test.mailzone.com -helo=05.spf1-test.mailzone.com result /.*/ fail smtp-comment /.*/ explanation header-comment /.*/ spfquery: domain of 05.spf1-test.mailzone.com does not designate 192.0.2.1 as permitted sender received-spf /.*/ Received-SPF: fail (spfquery: domain of 05.spf1-test.mailzone.com does not designate 192.0.2.1 as permitted sender) client-ip=192.0.2.1; firstname.lastname@example.org; helo=05.spf1-test.mailzone.com;
$ spfquery -ip=192.0.2.1 -sender=05.spf1-test.mailzone.com -helo=05.spf1-test.mailzone.com fail Please see http://www.openspf.org/why.html?sender=05.spf1-test.mailzone.com&ip=192.0.2.1&receiver=spfquery spfquery: domain of 05.spf1-test.mailzone.com does not designate 192.0.2.1 as permitted sender Received-SPF: fail (spfquery: domain of 05.spf1-test.mailzone.com does not designate 192.0.2.1 as permitted sender) client-ip=192.0.2.1; envelope-from=05.spf1-test.mailzone.com; helo=05.spf1-test.mailzone.com;
So, the standard implementation does what the test suite says.
Too bad that, if you bother checking the URL you are told to "please see"...
The domain 05.spf1-test.mailzone.com has published an SPF policy, however the policy is neutral on whether 192.0.2.1 is authorized to send mail on its behalf.
Either both the test suite and the sample implementation are wrong, or the site is wrong. And I am leaning towards "the test suite is wrong", because...
$ host -t txt 05.spf1-test.mailzone.com 05.spf1-test.mailzone.com descriptive text "v=spf1 default=deny"
If you check the record syntax (http://www.openspf.org/SPF_Record_Syntax) default is an unknown modifier, and should be ignored, so the record is simply "v=spf1", and indeed the result is neutral and there is no reason why this should be a fail.