Spec-like Tests in Python

CJ Gaconnet

www.gaconnet.com

Overview

Our Example - Verifying OAuth Signatures

>>> data = [('oauth_consumer_key', 'the-PFJ'),
            ('oauth_signature_method', 'HMAC-SHA1'),
            ('oauth_timestamp', 1323493200),
            ('oauth_nonce', 'thwow-him-to-the-floor')]

>>> qs = urlencode(sorted(data))
>>> url = quote('http://example.com/pilate/', safe='')
>>> base = 'GET&{url}&{qs}'.format(url=url, qs=qs)
>>> key = '{consumer}&'.format(consumer='secret')

>>> signature = hmac.new(key, base, sha1).digest()
>>> signature = quote(b64encode(signature), safe='')
>>> # 'N9l2SigcJUjORYMn9vGgJyjGP%2FI%3D'

Typical Python Tests - The Module

Django default:

# tests.py

If we're lucky:

# test_views.py

Typical Python Tests - The TestCase

class OAuthSignatureTestCase(TestCase):
    # ...

Typical Python Tests - The setUp

def setUp(self):
    self.user = create_test_api_user()
    self.url = 'http://example.com/pilate/'
    self.method = 'GET'

Typical Python Tests - The test_ methods

def test_verify(self):
    # ... all ``verify`` tests go here.

Typical Python Tests - The test_ method

def test_verify(self):
    oauth = dict(oauth_consumer_key=self.user.key,
                 oauth_signature_method=self.user.method,
                 oauth_timestamp=1323493200,
                 oauth_nonce='thwow-him-to-the-floor')

    # The happy path -- a valid signature
    request = dict(oauth)
    signature = sign(request, self.url, self.method,
                     self.user.secret)
    is_valid = verify(signature, request)
    self.assertTrue(is_valid)

Typical Python Tests - The test_ method

def test_verify(self):
    # oauth dict ... elided
    # The happy path ... elided

    # Timestamp too old
    request = dict(oauth, oauth_timestamp=100)
    signature = sign(request, self.url, self.method,
                     self.user.secret)
    is_valid = verify(signature, request)
    self.assertFalse(is_valid)

Typical Python Tests - The test_ method

def test_verify(self):
    # oauth dict ... elided
    # The happy path ... elided
    # Timestamp too old ... elided

    # Nonce already exists
    request = dict(oauth)
    save_nonce(timestamp=request['oauth_timestamp'],
               nonce=request['oauth_nonce'])
    signature = sign(request, self.url, self.method,
                     self.user.secret)
    is_valid = verify(signature, request)
    self.assertFalse(is_valid)

Typical Python Tests - The test_ method

def test_verify(self):
    # oauth dict ... elided
    # The happy path ... elided
    # Timestamp too old ... elided
    # Nonce already exists ... elided

    # Incorrect nonce in request
    signature = sign(oauth, self.url, self.method,
                     self.user.secret)
    request = dict(oauth, oauth_nonce='the-PFJ')
    is_valid = verify(signature, request)
    self.assertFalse(is_valid)

Typical Python Tests - The test_ method

def test_verify(self):
    # oauth dict ... elided
    # The happy path ... elided
    # Timestamp too old ... elided
    # Nonce already exists ... elided
    # Incorrect nonce in request ... elided

    # Empty signature
    signature = ''
    request = dict(oauth)
    is_valid = verify(signature, request)
    self.assertFalse(is_valid)

Typical Python Tests - The test_ method

def test_verify(self):
    # oauth dict ... elided
    # The happy path ... elided
    # Timestamp too old ... elided
    # Nonce already exists ... elided
    # Incorrect nonce in request ... elided
    # Empty signature ... elided

    # Extra param in request
    signature = sign(oauth, self.url, self.method,
                     self.user.secret)
    request = dict(oauth, extra='Manacles!')
    self.assertFalse(verify(signature, request))

Typical Python Tests - The test_ method

def test_verify(self):
    # oauth dict ... elided
    # The happy path ... elided
    # Timestamp too old ... elided
    # Nonce already exists ... elided
    # Incorrect nonce in request ... elided
    # Empty signature ... elided
    # Extra param in request ... elided

    # Missing nonce ...

Typical Python Tests - The test_ method

def test_verify(self):
    # oauth dict ... elided
    # The happy path ... elided
    # Timestamp too old ... elided
    # Nonce already exists ... elided
    # Incorrect nonce in request ... elided
    # Empty signature ... elided
    # Extra param in request ... elided
    # Missing nonce ... elided

    # Nonce exists for another client ...

Typical Python Tests - The test_ method

def test_verify(self):
    # oauth dict ... elided
    # The happy path ... elided
    # Timestamp too old ... elided
    # Nonce already exists ... elided
    # Incorrect nonce in request ... elided
    # Empty signature ... elided
    # Extra param in request ... elided
    # Missing nonce ... elided
    # Nonce exists for another client ... elided

    # and so on ...

Testing Pain - What caused failure?

=====================================================
FAIL: test_verify (test_views.OAuthSignatureTestCase)
-----------------------------------------------------
Traceback (most recent call last):
  File "tests/test_views.py", line 42, in test_verify
    self.assertFalse(is_valid)
AssertionError

-----------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

Testing Pain - What caused failure?

=====================================================
FAIL: test_verify (test_views.OAuthSignatureTestCase)
-----------------------------------------------------

Testing Pain - What caused failure?

  File "tests/test_views.py", line 42, in test_verify
    self.assertFalse(is_valid)
AssertionError

Testing Pain - Not DRY

request = dict(oauth)
signature = sign(request, self.url, self.method,
                 self.user.secret)
is_valid = verify(signature, request)

Repeated over and over

Testing Pain - Where is setUp?

def setUp(self):
    self.user = create_test_api_user()
    self.url = 'http://example.com/pilate/'
    self.method = 'GET'

What is boilerplate? What is relevant?

Testing Pain - Adding a new test

  1. Read all of test_verify
  2. Copy-paste one of the blocks
  3. Ensure failing assertion is the new one

Testing Pain - Review

Testing Pain - Takeaway

When a failing test makes us read 20+ lines of test code, we die inside.

Arrange Act Assert - The Testing Mantra

class MyTestCase(TestCase):
    def setUp(self):
        # Arrange generic

    def test_a_method(self):
        # Arrange specific

        # Act

        # Assert

        # repeat ...

Given When Then - The Behavior Mantra

Given When Then - The Behavior Mantra

Why?

User stories (specifications):

As a <role>, I want <feature>, so that <benefit>.

break down into scenarios (acceptance criteria):

Given <context>, when <event>, then <outcome>.

Given When Then - In Python's unittest?

Want

Given When Then - In Python's unittest?

class When___(TestCase):
    given = "People's Front of Judea"

    @property
    def given2(self):
        return "blessed-are-the-cheesemakers"

    def setUp(self):
        # Do "When" using "Given"s

    def test_it___(self):  # "Then"
        expected = "Thwow him to the floor again, sir?"
        self.assertEqual(self.result, expected)

Spec-like tests in Python - Given (generic)

class OAuthSignatureFixture(TestCase):
    signing_timestamp = 1000
    signing_nonce = 'blessed-are-the-cheesemakers'
    signing_key = 'the-PFJ'

    current_timestamp = 1000
    request_nonce = 'blessed-are-the-cheesemakers'
    extra = dict()

    @property
    def data(self):
        return dict(oauth_consumer_key=self.signing_key,
                    # ...

Spec-like tests in Python - When (generic)

class OAuthSignatureFixture(TestCase):
    # ... default "Given"s elided

    def save_nonce(self):
        """Override in a subclass to specialize"""
        pass

    def setUp(self):
        """Do "When" using "Given"s."""
        self.user = create_user()
        self.save_nonce()
        self.is_valid = verify(self.signature,
                               self.request)

Spec-like tests in Python - Scenarios

class WhenSignatureIsValid(OAuthSignatureFixture):
    def test_it_verifies(self):
        self.assertTrue(self.is_valid)

Spec-like tests in Python - Scenarios

class WhenTimestampIsTooOld(OAuthSignatureFixture):
    signature_timestamp = 10

    def test_it_does_not_verify(self):
        self.assertFalse(self.is_valid)

Spec-like tests in Python - Scenarios

class WhenNonceAlreadyExists(OAuthSignatureFixture):
    def save_nonce(self):
        # Create nonce record w/ `self.signing_nonce`
        # ...

    def test_it_does_not_verify(self):
        self.assertFalse(self.is_valid)

Spec-like tests in Python - Scenarios

class WhenRequestNonceDiffersFromSigningNonce(OAuthS#...
    signing_nonce = 'blessed-are-the-cheesemakers'
    request_nonce = 'the-PFJ'

    def test_it_does_not_verify(self):
        self.assertFalse(self.is_valid)

And so on...

Less Pain - What caused failure?

======================================================
FAIL: test_it_does_not_verify (test_signature.
                               WhenNonceAlreadyExists)
------------------------------------------------------
Traceback (most recent call last):
  File "tests/oauth/test_signature.py", line 42, in
  test_it_does_not_verify
    self.assertFalse(self.is_valid)
AssertionError
------------------------------------------------------
Ran 1 test in 0.001s
FAILED (failures=1)

Less Pain - What caused failure?

======================================================
FAIL: test_it_does_not_verify (test_signature.
                               WhenNonceAlreadyExists)
------------------------------------------------------

We may not even need to read the test's source.

Less Pain - More DRY

class WhenRequestIsMissingNonce(OAuthSignatureFixture):
    request_nonce = ''

    def test_it_does_not_verify(self):
        self.assertFalse(self.is_valid)

Less Pain - Adding a new test

  1. Skim all TestCase scenarios
  2. Copy-paste a similar one
  3. Edit specific givens
  4. Add any new test_ methods
class WhenSignatureIsEmpty(OAuthSignatureFixture):
    signature = ''

    def test_it_does_not_verify(self):
        self.assertFalse(self.is_valid)

Less Pain - Review

Caveats

References

  1. OAuth HMAC-SHA1
  2. test_dingus.py
  3. GoSpec example
  4. Nested Describes in Jasmine
  5. https://github.com/cucumber/cucumber/wiki/Given-When-Then
  6. What's in a Story?
  7. Breaking Down Features to User Stories
  8. Three Styles of Naming Tests
  9. Choice of Words in Testing Frameworks
  10. "Happy path" heard in DAS