Python unittest tips from JUnit view

This post is a track of tips by comparing unittest (previously named PyUnit) and JUnit. Most of the topics will also mention nose (nosetests) as the test runner and usually refer to the default or Parameterized runner of JUnit.

Origination:
- JUnit was created by Eric Gamma and Kent Beck originally for whitebox regression test. It initializes the XUnit family.
- unittest was originally the PyUnit as an XUnit implementation for Python whitebox testing.
Entrance point:
- Unittest starts from unittest.main(), while nose starts from nose.core.main, which is an alias of nose.core.TestProgram class.
- JUnit starts from junit.core.main().
Test Suite:
- This is similar, JUnit as TestSuite type to hold cases added from individual classes, unittest has unittest.TestSuite() to feed to runner.run($your_suite_instance).
- Python unittest is organized as Py packages and classes.
Hooks: Setup and Teardown:
- unittest picks setUp()/tearDown() as hooks for each test. And setUpClass()/tearDownClass() as @classmethod. Similarly, there are setUpModule()/tearDownModule() serve as global functions in module level.
- JUnit uses @Before and @After annotation from JUnit4. There are also @BeforeClass/@AfterClass static methods as class level hooks.
  - @Rules with TestName can be utilized to bypass or add special treatment to specific tests.
Timeout: (TBD)
Data Driven(Parameterized Test): (TBD)
IO Capture and Output Messages:
- Output messages:
  - Python nosetests inhibit stdout. With "-s" option, the stdout could be printed out. For messages generated by logging module, an option "--debug=$package_name_with_dot" can activate the logger output for specified package path.

Below is the CLI I used in my travis config YAML file. The python classes under test are located in me.maxwu. $packages and subdirectories.

> nosetests -s -v --logging-level=DEBUG --debug=me.maxwu --with-xunit --xunit-file circlestat_nose_xunit.xml --with-coverage --cover-package=me.maxwu --cover-html ./test

nosetests will capture stdout and store them into XUnit format test report when "-v" option is present. However, messages from logging will not persistent in XML format test report. Those logs could be persistented into logfiles with additional handlers defined in code. To hold the logs on CI cloud as travis and circle, we can specify the artifacts pattern in the configuration YAML file, ".travis.yml" and "circle.yml" (by default in root directory).

    - Java output messages

(TBD)

Skipped test:
- Python nose reads unittest.skip and notest decorator. However, the notest tagged methods are not "skipped", they are not reported in test report any more.

import unittest
@unittest.skip("temporarily disabled")
class XxxTest(unittest.TestCase):
    pass

import unittest
class XxxTest(unittest.TestCase):
    @unittest.skip("temporarily disabled")
    def test_Xxx(self):
        pass

JUnit uses @Ignore("reason") to bypass certain case methods.
Negative test:
- JUnit has three ways to capture expected Exception:
  - Use @Test(expected=$Exception.class)
  - Apply @Rule and implement an ExpectedException with thrown instance
  - Traditional try-catch and assert Exception info in catch block. Be careful to add fail() after try-catch.
- Resolving test gainst None with doctest:
  - doctest simple tests output string lines, which means None is just nothing in case expected value (not empty line with CR). For example, in below doctest, 4th case compares doctest capture to nothing (no result line) when the return value is expected as None.
  - To resolve it, we can just enhance the simple method call to an expression like ">>> func_call(...) is None" and easily compare(test) it against "True" as output value. the 5th and 6th case in below doctest shows how to fulfill it.

def casual_find_pos_with_bin(n, from_list):
   """Return the element position in **sorted** list or None.
   ...
   === Doc Test ===
   >>> casual_find_pos_with_bin(8, [x*2 for x in range(10)])
   4
   >>> casual_find_pos_with_bin(0, range(7))
   0
   >>> casual_find_pos_with_bin(6, range(7))
   6
   >>> casual_find_pos_with_bin(7, range(6))
   >>> casual_find_pos_with_bin(0, range(1,6)) is None
   >>> casual_find_pos_with_bin(8, [x*2+1 for x in range(6)]) is None
   >>> casual_find_pos_with_bin(4, [1,2,3,3,4,4,6])
   5
  	"""
  	# the method codes ...

Capturing expected exceptions with Python:

self.assertRaises(ExceptionClassName, function_name)

# or

with self.assertRaises($ExceptionClassName) as context:
	do_test_actions()
	
self.assertTrue("exception message" in context.exception)

Refer to python doc

[Updated by Feb11, Skipped/notest]

[Updated by Jan25, Initial sections]