selftest: add an expectedfail directory

We have some tests that are not only known to fail, but which are
intended to fail.

For example, to quote selftest/knownfail.d/dns:

> # These tests are expected to fail because we want to ensure that
> # unauthenticated updates are not permitted against the default
> # configuration, nor against an RODC

In contrast to selftest/knownfail.d/uac_objectclass_restrict, which
says:

> # All these tests need to be fixed and the entries here removed

That one should stay in selftest/knownfail.d.

Some files are mixed. For example, there are lines in
selftest/knownfail.d/smb1-tests which were added in *commits* that say

> We also need to add a knownfail (which will not be removed) for the
> new test which will fail in smb1 envs

but it is not clear to me that the whole file is expected to always
fail.

By moving some knownfails here, we allow selftest/knownfail.d to be a
bit more like a TODO list, containing things that actually constitute
failure.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

diff --git a/selftest/expectedfail.d/README b/selftest/expectedfail.d/README
new file mode 100644 (file)
index 0000000..5473e6c
--- /dev/null
+++ b/selftest/expectedfail.d/README
@@ -0,0 +1,22 @@
+# Files in this directory contain lists of regular expressions
+# matching the names of tests that are *necessarily* expected to fail.
+#
+# "make test" will not report failures for tests listed here and will
+# consider a successful run for any of these tests an error.
+#
+# They differ from the knownfail tests (selftest/knownfail.d) in that
+# the lack of failure here is definitely a problem. The knownfail
+# tests might be fixed one day, even accidentally, but these ones will
+# not.
+#
+# Before adding tests here, consider rewriting the test so that the
+# expected result is a failure. The tests in here are typically
+# testing the use of some protocol or feature on a server that has
+# that feature turned off. The same tests will also be run against
+# another server where they do not fail. The downside of this method
+# is we don't know that these expected fail tests are failing in the
+# right way.
+#
+# Empty lines and lines beginning with '#' are ignored.
+#
+# Please don't add tests to this README!

diff --git a/selftest/wscript b/selftest/wscript
index 785033e8ec728fe1378bc859fe028f08784c34db..daf497d5e62d48c04762708387862edea71fc62c 100644 (file)
--- a/selftest/wscript
+++ b/selftest/wscript
@@ -143,6 +143,7 @@ def cmd_testonly(opt):
     env.FILTER_XFAIL = ('${PYTHON} -u ${srcdir}/selftest/filter-subunit '
                         '--expected-failures=${srcdir}/selftest/knownfail '
                         '--expected-failures=${srcdir}/selftest/knownfail.d '
+                        '--expected-failures=${srcdir}/selftest/expectedfail.d '
                         '--flapping=${srcdir}/selftest/flapping '
                         '--flapping=${srcdir}/selftest/flapping.d')