From: Douglas Bagnall Date: Fri, 8 Dec 2023 03:05:36 +0000 (+1300) Subject: selftest: add an expectedfail directory X-Git-Url: http://git.samba.org/?a=commitdiff_plain;h=04ed1206057d7b08ae8c6270e1a6fcbc071e3844;p=scabrero%2Fsamba-autobuild%2F.git selftest: add an expectedfail directory We have some tests that are not only known to fail, but which are intended to fail. For example, to quote selftest/knownfail.d/dns: > # These tests are expected to fail because we want to ensure that > # unauthenticated updates are not permitted against the default > # configuration, nor against an RODC In contrast to selftest/knownfail.d/uac_objectclass_restrict, which says: > # All these tests need to be fixed and the entries here removed That one should stay in selftest/knownfail.d. Some files are mixed. For example, there are lines in selftest/knownfail.d/smb1-tests which were added in *commits* that say > We also need to add a knownfail (which will not be removed) for the > new test which will fail in smb1 envs but it is not clear to me that the whole file is expected to always fail. By moving some knownfails here, we allow selftest/knownfail.d to be a bit more like a TODO list, containing things that actually constitute failure. Signed-off-by: Douglas Bagnall Reviewed-by: Andrew Bartlett --- diff --git a/selftest/expectedfail.d/README b/selftest/expectedfail.d/README new file mode 100644 index 00000000000..5473e6cc9a1 --- /dev/null +++ b/selftest/expectedfail.d/README @@ -0,0 +1,22 @@ +# Files in this directory contain lists of regular expressions +# matching the names of tests that are *necessarily* expected to fail. +# +# "make test" will not report failures for tests listed here and will +# consider a successful run for any of these tests an error. +# +# They differ from the knownfail tests (selftest/knownfail.d) in that +# the lack of failure here is definitely a problem. The knownfail +# tests might be fixed one day, even accidentally, but these ones will +# not. +# +# Before adding tests here, consider rewriting the test so that the +# expected result is a failure. The tests in here are typically +# testing the use of some protocol or feature on a server that has +# that feature turned off. The same tests will also be run against +# another server where they do not fail. The downside of this method +# is we don't know that these expected fail tests are failing in the +# right way. +# +# Empty lines and lines beginning with '#' are ignored. +# +# Please don't add tests to this README! diff --git a/selftest/wscript b/selftest/wscript index 785033e8ec7..daf497d5e62 100644 --- a/selftest/wscript +++ b/selftest/wscript @@ -143,6 +143,7 @@ def cmd_testonly(opt): env.FILTER_XFAIL = ('${PYTHON} -u ${srcdir}/selftest/filter-subunit ' '--expected-failures=${srcdir}/selftest/knownfail ' '--expected-failures=${srcdir}/selftest/knownfail.d ' + '--expected-failures=${srcdir}/selftest/expectedfail.d ' '--flapping=${srcdir}/selftest/flapping ' '--flapping=${srcdir}/selftest/flapping.d')