4 Changes and improvements to testtools_, grouped by release.
12 * Failed equality assertions now line up. (Jonathan Lange, #879339)
18 This is a very big release. We've made huge improvements on three fronts:
19 1. Test failures are way nicer and easier to read
20 2. Matchers and ``assertThat`` are much more convenient to use
21 3. Correct handling of extended unicode characters
23 We've trimmed off the fat from the stack trace you get when tests fail, we've
24 cut out the bits of error messages that just didn't help, we've made it easier
25 to annotate mismatch failures, to compare complex objects and to match raised
28 Testing code was never this fun.
33 * ``AfterPreproccessing`` renamed to ``AfterPreprocessing``, which is a more
34 correct spelling. Old name preserved for backwards compatibility, but is
35 now deprecated. Please stop using it.
36 (Jonathan Lange, #813460)
38 * ``assertThat`` raises ``MismatchError`` instead of
39 ``TestCase.failureException``. ``MismatchError`` is a subclass of
40 ``AssertionError``, so in most cases this change will not matter. However,
41 if ``self.failureException`` has been set to a non-default value, then
42 mismatches will become test errors rather than test failures.
44 * ``gather_details`` takes two dicts, rather than two detailed objects.
45 (Jonathan Lange, #801027)
47 * ``MatchesRegex`` mismatch now says "<value> does not match /<regex>/" rather
48 than "<regex> did not match <value>". The regular expression contains fewer
49 backslashes too. (Jonathan Lange, #818079)
51 * Tests that run with ``AsynchronousDeferredRunTest`` now have the ``reactor``
52 attribute set to the running reactor. (Jonathan Lange, #720749)
57 * All public matchers are now in ``testtools.matchers.__all__``.
58 (Jonathan Lange, #784859)
60 * ``assertThat`` can actually display mismatches and matchers that contain
61 extended unicode characters. (Jonathan Lange, Martin [gz], #804127)
63 * ``assertThat`` output is much less verbose, displaying only what the mismatch
64 tells us to display. Old-style verbose output can be had by passing
65 ``verbose=True`` to assertThat. (Jonathan Lange, #675323, #593190)
67 * ``assertThat`` accepts a message which will be used to annotate the matcher.
68 This can be given as a third parameter or as a keyword parameter.
71 * Automated the Launchpad part of the release process.
72 (Jonathan Lange, #623486)
74 * Correctly display non-ASCII unicode output on terminals that claim to have a
75 unicode encoding. (Martin [gz], #804122)
77 * ``DocTestMatches`` correctly handles unicode output from examples, rather
78 than raising an error. (Martin [gz], #764170)
80 * ``ErrorHolder`` and ``PlaceHolder`` added to docs. (Jonathan Lange, #816597)
82 * ``ExpectedException`` now matches any exception of the given type by
83 default, and also allows specifying a ``Matcher`` rather than a mere regular
84 expression. (Jonathan Lange, #791889)
86 * ``FixtureSuite`` added, allows test suites to run with a given fixture.
89 * Hide testtools's own stack frames when displaying tracebacks, making it
90 easier for test authors to focus on their errors.
91 (Jonathan Lange, Martin [gz], #788974)
93 * Less boilerplate displayed in test failures and errors.
94 (Jonathan Lange, #660852)
96 * ``MatchesException`` now allows you to match exceptions against any matcher,
97 rather than just regular expressions. (Jonathan Lange, #791889)
99 * ``MatchesException`` now permits a tuple of types rather than a single type
100 (when using the type matching mode). (Robert Collins)
102 * ``MatchesStructure.byEquality`` added to make the common case of matching
103 many attributes by equality much easier. ``MatchesStructure.byMatcher``
104 added in case folk want to match by things other than equality.
107 * New convenience assertions, ``assertIsNone`` and ``assertIsNotNone``.
112 * ``AllMatch`` matches many values against a single matcher.
113 (Jonathan Lange, #615108)
115 * ``Contains``. (Robert Collins)
117 * ``GreaterThan``. (Christian Kampka)
119 * New helper, ``safe_hasattr`` added. (Jonathan Lange)
121 * ``reraise`` added to ``testtools.compat``. (Jonathan Lange)
127 This release brings consistent use of super for better compatibility with
128 multiple inheritance, fixed Python3 support, improvements in fixture and mather
129 outputs and a compat helper for testing libraries that deal with bytestrings.
134 * ``TestCase`` now uses super to call base ``unittest.TestCase`` constructor,
135 ``setUp`` and ``tearDown``. (Tim Cole, #771508)
137 * If, when calling ``useFixture`` an error occurs during fixture set up, we
138 still attempt to gather details from the fixture. (Gavin Panella)
144 * Additional compat helper for ``BytesIO`` for libraries that build on
145 testtools and are working on Python 3 porting. (Robert Collins)
147 * Corrected documentation for ``MatchesStructure`` in the test authors
148 document. (Jonathan Lange)
150 * ``LessThan`` error message now says something that is logically correct.
151 (Gavin Panella, #762008)
153 * Multiple details from a single fixture are now kept separate, rather than
154 being mooshed together. (Gavin Panella, #788182)
156 * Python 3 support now back in action. (Martin [gz], #688729)
158 * ``try_import`` and ``try_imports`` have a callback that is called whenever
159 they fail to import a module. (Martin Pool)
165 The last release of testtools could not be easy_installed. This is considered
166 severe enough for a re-release.
171 * Include ``doc/`` in the source distribution, making testtools installable
172 from PyPI again (Tres Seaver, #757439)
178 Many, many new matchers, vastly expanded documentation, stacks of bug fixes,
179 better unittest2 integration. If you've ever wanted to try out testtools but
180 been afraid to do so, this is the release to try.
186 * The timestamps generated by ``TestResult`` objects when no timing data has
187 been received are now datetime-with-timezone, which allows them to be
188 sensibly serialised and transported. (Robert Collins, #692297)
193 * ``AnnotatedMismatch`` now correctly returns details.
194 (Jonathan Lange, #724691)
196 * distutils integration for the testtools test runner. Can now use it for
197 'python setup.py test'. (Christian Kampka, #693773)
199 * ``EndsWith`` and ``KeysEqual`` now in testtools.matchers.__all__.
200 (Jonathan Lange, #692158)
202 * ``MatchesException`` extended to support a regular expression check against
203 the str() of a raised exception. (Jonathan Lange)
205 * ``MultiTestResult`` now forwards the ``time`` API. (Robert Collins, #692294)
207 * ``MultiTestResult`` now documented in the manual. (Jonathan Lange, #661116)
209 * New content helpers ``content_from_file``, ``content_from_stream`` and
210 ``attach_file`` make it easier to attach file-like objects to a
211 test. (Jonathan Lange, Robert Collins, #694126)
213 * New ``ExpectedException`` context manager to help write tests against things
214 that are expected to raise exceptions. (Aaron Bentley)
218 * ``MatchesListwise`` matches an iterable of matchers against an iterable
219 of values. (Michael Hudson-Doyle)
221 * ``MatchesRegex`` matches a string against a regular expression.
222 (Michael Hudson-Doyle)
224 * ``MatchesStructure`` matches attributes of an object against given
225 matchers. (Michael Hudson-Doyle)
227 * ``AfterPreproccessing`` matches values against a matcher after passing them
228 through a callable. (Michael Hudson-Doyle)
230 * ``MatchesSetwise`` matches an iterable of matchers against an iterable of
231 values, without regard to order. (Michael Hudson-Doyle)
233 * ``setup.py`` can now build a snapshot when Bazaar is installed but the tree
234 is not a Bazaar tree. (Jelmer Vernooij)
236 * Support for running tests using distutils (Christian Kampka, #726539)
238 * Vastly improved and extended documentation. (Jonathan Lange)
240 * Use unittest2 exception classes if available. (Jelmer Vernooij)
246 In this release we bring some very interesting improvements:
248 * new matchers for exceptions, sets, lists, dicts and more.
250 * experimental (works but the contract isn't supported) twisted reactor
253 * The built in runner can now list tests and filter tests (the -l and
254 --load-list options).
259 * addUnexpectedSuccess is translated to addFailure for test results that don't
260 know about addUnexpectedSuccess. Further, it fails the entire result for
261 all testtools TestResults (i.e. wasSuccessful() returns False after
262 addUnexpectedSuccess has been called). Note that when using a delegating
263 result such as ThreadsafeForwardingResult, MultiTestResult or
264 ExtendedToOriginalDecorator then the behaviour of addUnexpectedSuccess is
265 determined by the delegated to result(s).
266 (Jonathan Lange, Robert Collins, #654474, #683332)
268 * startTestRun will reset any errors on the result. That is, wasSuccessful()
269 will always return True immediately after startTestRun() is called. This
270 only applies to delegated test results (ThreadsafeForwardingResult,
271 MultiTestResult and ExtendedToOriginalDecorator) if the delegated to result
272 is a testtools test result - we cannot reliably reset the state of unknown
273 test result class instances. (Jonathan Lange, Robert Collins, #683332)
275 * Responsibility for running test cleanups has been moved to ``RunTest``.
276 This change does not affect public APIs and can be safely ignored by test
277 authors. (Jonathan Lange, #662647)
284 * ``EndsWith`` which complements the existing ``StartsWith`` matcher.
285 (Jonathan Lange, #669165)
287 * ``MatchesException`` matches an exception class and parameters. (Robert
290 * ``KeysEqual`` matches a dictionary with particular keys. (Jonathan Lange)
292 * ``assertIsInstance`` supports a custom error message to be supplied, which
293 is necessary when using ``assertDictEqual`` on Python 2.7 with a
294 ``testtools.TestCase`` base class. (Jelmer Vernooij)
296 * Experimental support for running tests that return Deferreds.
297 (Jonathan Lange, Martin [gz])
299 * Provide a per-test decorator, run_test_with, to specify which RunTest
300 object to use for a given test. (Jonathan Lange, #657780)
302 * Fix the runTest parameter of TestCase to actually work, rather than raising
303 a TypeError. (Jonathan Lange, #657760)
305 * Non-release snapshots of testtools will now work with buildout.
306 (Jonathan Lange, #613734)
308 * Malformed SyntaxErrors no longer blow up the test suite. (Martin [gz])
310 * ``MismatchesAll.describe`` no longer appends a trailing newline.
311 (Michael Hudson-Doyle, #686790)
313 * New helpers for conditionally importing modules, ``try_import`` and
314 ``try_imports``. (Jonathan Lange)
316 * ``Raises`` added to the ``testtools.matchers`` module - matches if the
317 supplied callable raises, and delegates to an optional matcher for validation
318 of the exception. (Robert Collins)
320 * ``raises`` added to the ``testtools.matchers`` module - matches if the
321 supplied callable raises and delegates to ``MatchesException`` to validate
322 the exception. (Jonathan Lange)
324 * Tests will now pass on Python 2.6.4 : an ``Exception`` change made only in
325 2.6.4 and reverted in Python 2.6.5 was causing test failures on that version.
326 (Martin [gz], #689858).
328 * ``testtools.TestCase.useFixture`` has been added to glue with fixtures nicely.
331 * ``testtools.run`` now supports ``-l`` to list tests rather than executing
332 them. This is useful for integration with external test analysis/processing
333 tools like subunit and testrepository. (Robert Collins)
335 * ``testtools.run`` now supports ``--load-list``, which takes a file containing
336 test ids, one per line, and intersects those ids with the tests found. This
337 allows fine grained control of what tests are run even when the tests cannot
338 be named as objects to import (e.g. due to test parameterisation via
339 testscenarios). (Robert Collins)
341 * Update documentation to say how to use testtools.run() on Python 2.4.
342 (Jonathan Lange, #501174)
344 * ``text_content`` conveniently converts a Python string to a Content object.
345 (Jonathan Lange, James Westby)
352 Lots of little cleanups in this release; many small improvements to make your
353 testing life more pleasant.
358 * Cleanups can raise ``testtools.MultipleExceptions`` if they have multiple
359 exceptions to report. For instance, a cleanup which is itself responsible for
360 running several different internal cleanup routines might use this.
362 * Code duplication between assertEqual and the matcher Equals has been removed.
364 * In normal circumstances, a TestCase will no longer share details with clones
365 of itself. (Andrew Bennetts, bug #637725)
367 * Less exception object cycles are generated (reduces peak memory use between
368 garbage collection). (Martin [gz])
370 * New matchers 'DoesNotStartWith' and 'StartsWith' contributed by Canonical
371 from the Launchpad project. Written by James Westby.
373 * Timestamps as produced by subunit protocol clients are now forwarded in the
374 ThreadsafeForwardingResult so correct test durations can be reported.
375 (Martin [gz], Robert Collins, #625594)
377 * With unittest from Python 2.7 skipped tests will now show only the reason
378 rather than a serialisation of all details. (Martin [gz], #625583)
380 * The testtools release process is now a little better documented and a little
381 smoother. (Jonathan Lange, #623483, #623487)
387 Nothing major in this release, just enough small bits and pieces to make it
388 useful enough to upgrade to.
390 In particular, a serious bug in assertThat() has been fixed, it's easier to
391 write Matchers, there's a TestCase.patch() method for those inevitable monkey
392 patches and TestCase.assertEqual gives slightly nicer errors.
397 * 'TestCase.assertEqual' now formats errors a little more nicely, in the
400 * Added `PlaceHolder` and `ErrorHolder`, TestCase-like objects that can be
401 used to add results to a `TestResult`.
403 * 'Mismatch' now takes optional description and details parameters, so
404 custom Matchers aren't compelled to make their own subclass.
406 * jml added a built-in UTF8_TEXT ContentType to make it slightly easier to
407 add details to test results. See bug #520044.
409 * Fix a bug in our built-in matchers where assertThat would blow up if any
410 of them failed. All built-in mismatch objects now provide get_details().
412 * New 'Is' matcher, which lets you assert that a thing is identical to
415 * New 'LessThan' matcher which lets you assert that a thing is less than
418 * TestCase now has a 'patch()' method to make it easier to monkey-patching
419 objects in tests. See the manual for more information. Fixes bug #310770.
421 * MultiTestResult methods now pass back return values from the results it
427 This release fixes some obscure traceback formatting issues that probably
428 weren't affecting you but were certainly breaking our own test suite.
433 * Jamu Kakar has updated classes in testtools.matchers and testtools.runtest
434 to be new-style classes, fixing bug #611273.
439 * Martin[gz] fixed traceback handling to handle cases where extract_tb returns
440 a source line of None. Fixes bug #611307.
442 * Martin[gz] fixed an unicode issue that was causing the tests to fail,
445 * testtools now handles string exceptions (although why would you want to use
446 them?) and formats their tracebacks correctly. Thanks to Martin[gz] for
452 This release overhauls the traceback formatting layer to deal with Python 2
453 line numbers and traceback objects often being local user encoded strings
454 rather than unicode objects. Test discovery has also been added and Python 3.1
455 is also supported. Finally, the Mismatch protocol has been extended to let
456 Matchers collaborate with tests in supplying detailed data about failures.
461 * testtools.utils has been renamed to testtools.compat. Importing
462 testtools.utils will now generate a deprecation warning.
467 * Add machinery for Python 2 to create unicode tracebacks like those used by
468 Python 3. This means testtools no longer throws on encountering non-ascii
469 filenames, source lines, or exception strings when displaying test results.
470 Largely contributed by Martin[gz] with some tweaks from Robert Collins.
472 * James Westby has supplied test discovery support using the Python 2.7
473 TestRunner in testtools.run. This requires the 'discover' module. This
476 * Python 3.1 is now supported, thanks to Martin[gz] for a partial patch.
477 This fixes bug #592375.
479 * TestCase.addCleanup has had its docstring corrected about when cleanups run.
481 * TestCase.skip is now deprecated in favour of TestCase.skipTest, which is the
482 Python2.7 spelling for skip. This closes bug #560436.
484 * Tests work on IronPython patch from Martin[gz] applied.
486 * Thanks to a patch from James Westby testtools.matchers.Mismatch can now
487 supply a get_details method, which assertThat will query to provide
488 additional attachments. This can be used to provide additional detail
489 about the mismatch that doesn't suite being included in describe(). For
490 instance, if the match process was complex, a log of the process could be
491 included, permitting debugging.
493 * testtools.testresults.real._StringException will now answer __str__ if its
494 value is unicode by encoding with UTF8, and vice versa to answer __unicode__.
495 This permits subunit decoded exceptions to contain unicode and still format
501 More matchers, Python 2.4 support, faster test cloning by switching to copy
502 rather than deepcopy and better output when exceptions occur in cleanups are
503 the defining characteristics of this release.
508 * New matcher "Annotate" that adds a simple string message to another matcher,
509 much like the option 'message' parameter to standard library assertFoo
512 * New matchers "Not" and "MatchesAll". "Not" will invert another matcher, and
513 "MatchesAll" that needs a successful match for all of its arguments.
515 * On Python 2.4, where types.FunctionType cannot be deepcopied, testtools will
516 now monkeypatch copy._deepcopy_dispatch using the same trivial patch that
517 added such support to Python 2.5. The monkey patch is triggered by the
518 absence of FunctionType from the dispatch dict rather than a version check.
521 * On windows the test 'test_now_datetime_now' should now work reliably.
523 * TestCase.getUniqueInteger and TestCase.getUniqueString now have docstrings.
525 * TestCase.getUniqueString now takes an optional prefix parameter, so you can
526 now use it in circumstances that forbid strings with '.'s, and such like.
528 * testtools.testcase.clone_test_with_new_id now uses copy.copy, rather than
529 copy.deepcopy. Tests that need a deeper copy should use the copy protocol to
530 control how they are copied. Bug #498869.
532 * The backtrace test result output tests should now pass on windows and other
533 systems where os.sep is not '/'.
535 * When a cleanUp or tearDown exception occurs, it is now accumulated as a new
536 traceback in the test details, rather than as a separate call to addError /
537 addException. This makes testtools work better with most TestResult objects
538 and fixes bug #335816.
544 Python 3 support, more matchers and better consistency with Python 2.7 --
545 you'd think that would be enough for a point release. Well, we here on the
546 testtools project think that you deserve more.
548 We've added a hook so that user code can be called just-in-time whenever there
549 is an exception, and we've also factored out the "run" logic of test cases so
550 that new outcomes can be added without fiddling with the actual flow of logic.
552 It might sound like small potatoes, but it's changes like these that will
553 bring about the end of test frameworks.
559 * A failure in setUp and tearDown now report as failures not as errors.
561 * Cleanups now run after tearDown to be consistent with Python 2.7's cleanup
564 * ExtendedToOriginalDecorator now passes unrecognised attributes through
565 to the decorated result object, permitting other extensions to the
566 TestCase -> TestResult protocol to work.
568 * It is now possible to trigger code just-in-time after an exception causes
569 a test outcome such as failure or skip. See the testtools MANUAL or
570 ``pydoc testtools.TestCase.addOnException``. (bug #469092)
572 * New matcher Equals which performs a simple equality test.
574 * New matcher MatchesAny which looks for a match of any of its arguments.
576 * TestCase no longer breaks if a TestSkipped exception is raised with no
579 * TestCase.run now clones test cases before they are run and runs the clone.
580 This reduces memory footprint in large test runs - state accumulated on
581 test objects during their setup and execution gets freed when test case
582 has finished running unless the TestResult object keeps a reference.
583 NOTE: As test cloning uses deepcopy, this can potentially interfere if
584 a test suite has shared state (such as the testscenarios or testresources
585 projects use). Use the __deepcopy__ hook to control the copying of such
586 objects so that the shared references stay shared.
588 * Testtools now accepts contributions without copyright assignment under some
589 circumstances. See HACKING for details.
591 * Testtools now provides a convenient way to run a test suite using the
592 testtools result object: python -m testtools.run testspec [testspec...].
594 * Testtools now works on Python 3, thanks to Benjamin Peterson.
596 * Test execution now uses a separate class, testtools.RunTest to run single
597 tests. This can be customised and extended in a more consistent fashion than
598 the previous run method idiom. See pydoc for more information.
600 * The test doubles that testtools itself uses are now available as part of
601 the testtools API in testtols.testresult.doubles.
603 * TracebackContent now sets utf8 as the charset encoding, rather than not
604 setting one and encoding with the default encoder.
606 * With python2.7 testtools.TestSkipped will be the unittest.case.SkipTest
607 exception class making skips compatible with code that manually raises the
608 standard library exception. (bug #490109)
613 * TestCase.getUniqueInteger is now implemented using itertools.count. Thanks
614 to Benjamin Peterson for the patch. (bug #490111)
620 The new matcher API introduced in 0.9.0 had a small flaw where the matchee
621 would be evaluated twice to get a description of the mismatch. This could lead
622 to bugs if the act of matching caused side effects to occur in the matchee.
623 Since having such side effects isn't desirable, we have changed the API now
624 before it has become widespread.
629 * Matcher API changed to avoid evaluating matchee twice. Please consult
630 the API documentation.
632 * TestCase.getUniqueString now uses the test id, not the test method name,
633 which works nicer with parameterised tests.
638 * Python2.4 is now supported again.
644 This release of testtools is perhaps the most interesting and exciting one
645 it's ever had. We've continued in bringing together the best practices of unit
646 testing from across a raft of different Python projects, but we've also
647 extended our mission to incorporating unit testing concepts from other
648 languages and from our own research, led by Robert Collins.
650 We now support skipping and expected failures. We'll make sure that you
651 up-call setUp and tearDown, avoiding unexpected testing weirdnesses. We're
652 now compatible with Python 2.5, 2.6 and 2.7 unittest library.
654 All in all, if you are serious about unit testing and want to get the best
655 thinking from the whole Python community, you should get this release.
660 * A new TestResult API has been added for attaching details to test outcomes.
661 This API is currently experimental, but is being prepared with the intent
662 of becoming an upstream Python API. For more details see pydoc
663 testtools.TestResult and the TestCase addDetail / getDetails methods.
665 * assertThat has been added to TestCase. This new assertion supports
666 a hamcrest-inspired matching protocol. See pydoc testtools.Matcher for
667 details about writing matchers, and testtools.matchers for the included
668 matchers. See http://code.google.com/p/hamcrest/.
670 * Compatible with Python 2.6 and Python 2.7
672 * Failing to upcall in setUp or tearDown will now cause a test failure.
673 While the base methods do nothing, failing to upcall is usually a problem
674 in deeper hierarchies, and checking that the root method is called is a
675 simple way to catch this common bug.
677 * New TestResult decorator ExtendedToOriginalDecorator which handles
678 downgrading extended API calls like addSkip to older result objects that
679 do not support them. This is used internally to make testtools simpler but
680 can also be used to simplify other code built on or for use with testtools.
682 * New TextTestResult supporting the extended APIs that testtools provides.
684 * Nose will no longer find 'runTest' tests in classes derived from
685 testtools.testcase.TestCase (bug #312257).
687 * Supports the Python 2.7/3.1 addUnexpectedSuccess and addExpectedFailure
688 TestResult methods, with a support function 'knownFailure' to let tests
689 trigger these outcomes.
691 * When using the skip feature with TestResult objects that do not support it
692 a test success will now be reported. Previously an error was reported but
693 production experience has shown that this is too disruptive for projects that
694 are using skips: they cannot get a clean run on down-level result objects.
697 .. _testtools: http://pypi.python.org/pypi/testtools