Meta-Testing your Assertions

As gems get more complex it is good practice to ship additional assertion methods with your library, so users can easily test code they wrote using your library.

Usually you do this by providing either a module or some derived TestCase, in a Test::Unit or MiniTest environment.

Tests without special assertions

The world-famous are-u-cool gem checks persons – whether they’re cool or not by querying some awesome web service.

Applications using this gem would usually have some test like this.

class PersonsTest < Unit::TestCase
  it "is the opposite of cool" do

Domain-specific tests

Since this is a lot of work, the gem ships with its own test case to help you.

class AreUCool::TestCase < Unit::TestCase
  def setup
    @instance =
  # Asserts +name+ is cool.
  def assert_cool(name)

Simplifying your application tests, an example usage would naturally look like the following.

class PersonsTest < AreUCool::TestCase
  it "is extremely cool" do
    assert_cool "Nick"

See the difference?

The new test case makes it easy to test coolness.

Testing your test

Many gems do this. They provide modules or test classes to make your life easier.

However, mostly these test extensions aren’t tested at all, or are tested wrong. The gem authors usually “test” their test methods like this (as seen in Rails and in my own tests).

# in are-u-cool/test/test_case_test.rb
class TestCaseTest < AreUCool::TestCase
  it "should respond to assert_cool" do
    assert_cool "Nick"

The problem here is: The tested test tests itself, which is almost like this unit test, for instance.

class PersonsTest < Test::Unit
  it "should respond to #name" do

This test just tests if a method can be run in general. The same happens in the second last example. We use the new assertion, nothing more.

Don’t mix meta levels

We confused two different layers here! So, keep these rules in mind when testing your tests:

  • Don’t derive your test case from that which you’re testing!
  • Double-assert your new assertions!

Testing a TestCase is relatively simple. Look at how the new TestCaseTest looks.

class TestCaseTest < Unit::TestCase
  setup do
    @test =
  it "should respond to assert_cool" do
    assert @test.assert_cool("Nick")

Notice how I use two asserts in one line? This cleanly tests if the new assertion returns true on a 100% valid assertion.

Being true pessimists, we double-check a failing assertion, too.

  it "should complain about uncool persons" do
    exc = assert_raises(MiniTest::Assertion) do

    assert_equal "Expected true.", exc.message

We simply catch the Assertion exception that is thrown if the assert_cool assertion fails.

Now, how would you do that in a self-contained test? You wouldn’t, right.

Following this, you should never use a test to test itself, but run tests against an instance of it – cleanly separating meta levels.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s