Günther Deschner [Mon, 24 Oct 2022 13:42:54 +0000 (15:42 +0200)]
s3-vfs_fruit: add 'fruit:veto_localized' option
This new boolean option allows to automatically treat ".localized" as
vetoed files. Any attempts to access a ".localized" file or directory
will result in an access failure. MacOS tries to check for the presence
of such a file to check for the need of localized name display in
directory listings or checks for .localized directories containing name
translations). In large directory listings, typically half of the
compound SMB2 queries are done for ".localized", causing a significant
performance impact in particular with clustered filesystems.
More information can be found here:
https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemAdvancedPT/LocalizingtheNameofaDirectory/LocalizingtheNameofaDirectory.html
Guenther
Signed-off-by: Guenther Deschner <gd@samba.org>
Günther Deschner [Wed, 9 Nov 2022 15:21:16 +0000 (16:21 +0100)]
s4-auth: fix sam test binary ntstatus include path
Guenther
Signed-off-by: Guenther Deschner <gd@samba.org>
Volker Lendecke [Sat, 3 Dec 2022 20:43:06 +0000 (21:43 +0100)]
smbd: Close the opened file in smbd_smb2_create_after_exec() error case
smbd_smb2_create_after_exec() is only called when the file has
successfully been opened. When this fails in the middle, we can't
leave the fsp around. Hard to test with current code, but with reparse
point handling we'll have a reproducable case soon.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Dec 6 23:37:52 UTC 2022 on sn-devel-184
Volker Lendecke [Sat, 3 Dec 2022 20:42:49 +0000 (21:42 +0100)]
smbd: Centralize error handling in smbd_smb2_create_after_exec()
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Nicolas Williams [Wed, 10 Mar 2021 22:49:04 +0000 (16:49 -0600)]
CVE-2022-44640 HEIMDAL: asn1: invalid free in ASN.1 codec
Heimdal's ASN.1 compiler generates code that allows specially
crafted DER encodings of CHOICEs to invoke the wrong free function
on the decoded structure upon decode error. This is known to impact
the Heimdal KDC, leading to an invalid free() of an address partly
or wholly under the control of the attacker, in turn leading to a
potential remote code execution (RCE) vulnerability.
This error affects the DER codec for all CHOICE types used in
Heimdal, though not all cases will be exploitable. We have not
completed a thorough analysis of all the Heimdal components
affected, thus the Kerberos client, the X.509 library, and other
parts, may be affected as well.
This bug has been in Heimdal since 2005. It was first reported by
Douglas Bagnall, though it had been found independently by the
Heimdal maintainers via fuzzing a few weeks earlier.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14929
(cherry-picked from Heimdal commit
9c9dac2b169255bad9071eea99fa90b980dde767)
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Tue Dec 6 13:41:05 UTC 2022 on sn-devel-184
Andrew Bartlett [Tue, 6 Dec 2022 02:11:05 +0000 (15:11 +1300)]
CVE-2022-44640 selftest: Exclude Heimdal fuzz-inputs from source_chars test
A new file will shorlty fail as it is binary input
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14929
Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Anoop C S [Mon, 5 Dec 2022 10:54:46 +0000 (16:24 +0530)]
lib/compression: Include missing stat header file
<sys/stat.h> was missing from compression library tests which resulted
in the following compile time error:
../../lib/compression/tests/test_lzx_huffman.c: In function
‘datablob_from_file’:
../../lib/compression/tests/test_lzx_huffman.c:383:21: error:
storage size of ‘s’ isn’t known
383 | struct stat s;
| ^
../../lib/compression/tests/test_lzx_huffman.c:389:15: warning:
implicit declaration of function ‘fstat’ [-Wimplicit-function-declaration]
389 | ret = fstat(fileno(fh), &s);
| ^~~~~
Signed-off-by: Anoop C S <anoopcs@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Tue Dec 6 11:39:16 UTC 2022 on sn-devel-184
Noel Power [Fri, 25 Nov 2022 13:04:17 +0000 (13:04 +0000)]
python/samba: use s3 param samba config parsing
follup to commit:
b4d7540bb4798e6801accf34a26fc0f2636bdd1f
fix another instance to use s3 config parsing which is more
forgiving (e.g. include directives that point to non existing
files are ignored)
Signed-off-by: Noel Power <npower@samba.org>
Reviewed-by: David Mulder <dmulder@samba.org>
Autobuild-User(master): Noel Power <npower@samba.org>
Autobuild-Date(master): Tue Dec 6 10:38:56 UTC 2022 on sn-devel-184
Volker Lendecke [Sun, 4 Dec 2022 11:16:39 +0000 (12:16 +0100)]
smbd: Simplify symlink_target_below_conn()
readlink_talloc() deals exactly the same way with a NULL relname
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Mon Dec 5 16:06:51 UTC 2022 on sn-devel-184
Volker Lendecke [Sun, 4 Dec 2022 11:14:12 +0000 (12:14 +0100)]
smbd: Simplify readlink_talloc()
SMB_VFS_READLINKAT() just looks at the basename, we can avoid the
relname being talloc'ed
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Volker Lendecke [Mon, 24 Oct 2022 17:56:31 +0000 (19:56 +0200)]
smbd: No dfs_filename_convert() in filename_convert_smb1_search_path()
We further down call filename_convert_dirfsp(), which also has this
call. No need to copy that code here as well.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Volker Lendecke [Sun, 4 Dec 2022 10:07:09 +0000 (11:07 +0100)]
libsmb: Remove sync cli_posix_readlink() wrapper
cli_readlink() now covers smb1 posix extensions as well
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Volker Lendecke [Wed, 12 Oct 2022 18:38:14 +0000 (20:38 +0200)]
smbclient: Use cli_readlink
Make smbclient's readlink command also work for SMB2 reparse style
symlink.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Volker Lendecke [Wed, 12 Oct 2022 18:35:10 +0000 (20:35 +0200)]
libsmb: Make readlink issue posix_readlink
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Volker Lendecke [Tue, 11 Oct 2022 15:01:28 +0000 (17:01 +0200)]
smbd: Fix a comment
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Volker Lendecke [Wed, 12 Oct 2022 05:27:36 +0000 (07:27 +0200)]
smbd: Slightly simplify smb_posix_unlink()
We did check VALID_STAT() above.
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Volker Lendecke [Fri, 2 Dec 2022 09:34:55 +0000 (10:34 +0100)]
tests: Test error codes for SET_REPARSE_POINT
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: David Mulder <dmulder@samba.org>
Volker Lendecke [Fri, 2 Dec 2022 09:20:06 +0000 (10:20 +0100)]
tests: Try setting a 0-sized reparse point
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: David Mulder <dmulder@samba.org>
Volker Lendecke [Fri, 2 Dec 2022 09:17:15 +0000 (10:17 +0100)]
tests: Ignore symlink trusts flags in symlink error returns
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: David Mulder <dmulder@samba.org>
Volker Lendecke [Fri, 2 Dec 2022 09:10:12 +0000 (10:10 +0100)]
pylibsmb: Add symlink flags
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: David Mulder <dmulder@samba.org>
Volker Lendecke [Fri, 2 Dec 2022 09:06:31 +0000 (10:06 +0100)]
lib: Add symlink trust flags from dochelp
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: David Mulder <dmulder@samba.org>
Volker Lendecke [Fri, 2 Dec 2022 08:26:56 +0000 (09:26 +0100)]
tests: Fix use of self.assertRaises()
The with statement creates a new variable. I thought it opens a block
where "e" is only valid in that block. But instead it runs the whole
thing, expecting an exception somewhere. Learning python....
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: David Mulder <dmulder@samba.org>
Volker Lendecke [Thu, 1 Dec 2022 14:14:03 +0000 (15:14 +0100)]
tests: Show that we can write to a reparse point file
Works against Windows 2016
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: David Mulder <dmulder@samba.org>
Volker Lendecke [Thu, 1 Dec 2022 13:49:37 +0000 (14:49 +0100)]
tests: Show that a directory with a reparse point can't be populated
Works against Windows 2016
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: David Mulder <dmulder@samba.org>
Volker Lendecke [Thu, 1 Dec 2022 13:48:46 +0000 (14:48 +0100)]
tests: IO_REPARSE_TAG_NOT_HANDLED is acceptable for unlink
This happens when a path has an unknown reparse point in the middle
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: David Mulder <dmulder@samba.org>
Andreas Schneider [Mon, 5 Dec 2022 10:18:10 +0000 (11:18 +0100)]
s3:utils: Fix stack smashing in net offlinejoin
Cast from 'uint32_t *' (aka 'unsigned int *') to 'size_t *' (aka
'unsigned long *') increases required alignment from 4 to 8
==10343==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffdc6784fc0 at pc 0x7f339f1ea500 bp 0x7ffdc6784ed0 sp 0x7ffdc6784ec8
WRITE of size 8 at 0x7ffdc6784fc0 thread T0
#0 0x7f339f1ea4ff in fd_load ../../lib/util/util_file.c:220
#1 0x7f339f1ea5a4 in file_load ../../lib/util/util_file.c:245
#2 0x56363209a596 in net_offlinejoin_requestodj ../../source3/utils/net_offlinejoin.c:267
#3 0x56363209a9d0 in net_offlinejoin ../../source3/utils/net_offlinejoin.c:74
#4 0x56363208f61c in net_run_function ../../source3/utils/net_util.c:453
#5 0x563631fe8a9f in main ../../source3/utils/net.c:1358
#6 0x7f339b22c5af in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#7 0x7f339b22c678 in __libc_start_main_impl ../csu/libc-start.c:381
#8 0x563631faf374 in _start ../sysdeps/x86_64/start.S:115
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15257
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Mon Dec 5 12:05:24 UTC 2022 on sn-devel-184
Andreas Schneider [Thu, 1 Dec 2022 14:49:43 +0000 (15:49 +0100)]
nsswitch:tests: Use ldb(modify|search) from the system
If Samba is built against the system libldb, use the system tools.
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Mon Dec 5 09:36:40 UTC 2022 on sn-devel-184
Mikhail Novosyolov [Fri, 21 Oct 2022 09:08:39 +0000 (12:08 +0300)]
manpages: samba-dcerpcd: fix typo (add missing space)
Signed-off-by: Mikhail Novosyolov <m.novosyolov@rosalinux.ru>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Andreas Schneider [Sat, 3 Dec 2022 17:06:43 +0000 (18:06 +0100)]
testprogs: Do not run tests if undump.sh is not available
We don't include source4/selftest/provisions/ in source tarballs!
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Mon Dec 5 08:22:29 UTC 2022 on sn-devel-184
Andreas Schneider [Sun, 4 Dec 2022 18:46:36 +0000 (19:46 +0100)]
testprogs: If built against system db use the system tools in ldapcmp_restoredc.sh
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Andreas Schneider [Sun, 4 Dec 2022 18:44:52 +0000 (19:44 +0100)]
testprogs: If built against system db use the system tools in test_net_ads_dns.sh
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Andreas Schneider [Sun, 4 Dec 2022 18:34:35 +0000 (19:34 +0100)]
testprogs: If built against system db use the system tools in test_trust_token.sh
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Andreas Schneider [Sat, 3 Dec 2022 16:48:33 +0000 (17:48 +0100)]
testprogs: If built against system db use the system tools in test_primary_group.sh
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Andreas Schneider [Sat, 3 Dec 2022 17:20:53 +0000 (18:20 +0100)]
lib:compression: Initialize variables
lib/compression/tests/test_lzx_huffman.c: In function ‘test_lzxpress_huffman_overlong_matches’:
lib/compression/tests/test_lzx_huffman.c:1013:35: error: ‘j’ may be used uninitialized [-Werror=maybe-uninitialized]
1013 | assert_int_equal(score, i * j);
| ^
lib/compression/tests/test_lzx_huffman.c:979:19: note: ‘j’ was declared here
979 | size_t i, j;
| ^
lib/compression/tests/test_lzx_huffman.c: In function ‘test_lzxpress_huffman_overlong_matches_abc’:
lib/compression/tests/test_lzx_huffman.c:1059:39: error: ‘k’ may be used uninitialized [-Werror=maybe-uninitialized]
1059 | assert_int_equal(score, i * j * k);
| ^
lib/compression/tests/test_lzx_huffman.c:1020:22: note: ‘k’ was declared here
1020 | size_t i, j, k;
| ^
lib/compression/tests/test_lzx_huffman.c:1059:35: error: ‘j’ may be used uninitialized [-Werror=maybe-uninitialized]
1059 | assert_int_equal(score, i * j * k);
| ^
lib/compression/tests/test_lzx_huffman.c:1020:19: note: ‘j’ was declared here
1020 | size_t i, j, k;
| ^
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Sun Dec 4 09:12:30 UTC 2022 on sn-devel-184
Christof Schmitt [Tue, 29 Nov 2022 23:51:10 +0000 (16:51 -0700)]
nfs4_acl: Add comment for setting ACL as root
Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Fri Dec 2 08:02:13 UTC 2022 on sn-devel-184
Christof Schmitt [Tue, 12 Jul 2022 23:35:37 +0000 (16:35 -0700)]
posix_acls: Make try_chown and unpack_nt_owners static
These functions are now only called from check_chown in posix_acls.c
Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Christof Schmitt [Tue, 12 Jul 2022 23:32:08 +0000 (16:32 -0700)]
nfs4_acls: Call chown_if_needed function to remove duplicate code
Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Christof Schmitt [Tue, 29 Nov 2022 23:46:24 +0000 (16:46 -0700)]
posix_acl: Move chown checks to new function
Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Christof Schmitt [Tue, 12 Jul 2022 23:08:07 +0000 (16:08 -0700)]
posix_acls: Remove redundant call to save mode
The same assignment is already done earlier, and nothing is changed in
between.
Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Douglas Bagnall [Thu, 24 Nov 2022 23:46:08 +0000 (12:46 +1300)]
lib/compression/lzxpress: fix our slow compression
This uses the same hash table method as lzxpress_huffman, though the
code can't be directly reused as the sizes of the offsets is
different, and there is not a block processing step here.
This will worsen the compression ratio compared to the exhaustive
search we previously used, though we still perform better than
Windows. To put numbers on it, the test files used to compress to 0.91
of Windows' compression size, and now they compress to 0.96.
On the other hand this is many orders of magnitude faster. It is
difficult to say exactly how much faster -- while the testsuite time
has only improved 200-fold (from 7 minutes to 2 seconds), most of the
remaining 2 seconds is used in data generation and management, not
compression. OSSFuzz consistently finds new vectors that time out
after a minute; on these we'll see nearly an order of magnitude of
orders of magnitude inprovement.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Autobuild-User(master): Joseph Sutton <jsutton@samba.org>
Autobuild-Date(master): Fri Dec 2 00:00:04 UTC 2022 on sn-devel-184
Douglas Bagnall [Thu, 24 Nov 2022 23:38:11 +0000 (12:38 +1300)]
lib/compression/lzxpress: shift encoding into helper functions
This makes it easier to rework the encoding decision to depend on a
hash table match rather than the current exhaustive search.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 03:15:00 +0000 (16:15 +1300)]
lib/compression/lzxpress compression: use a write context struct
This will make it possible to move encoding operations into helper
functions, which will make it easier to restructure the code to use a
hash table for faster matching.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Wed, 23 Nov 2022 22:44:35 +0000 (11:44 +1300)]
lib/compression: more tests for lzxpress plain compression
These are based on (i.e. copied and pasted from) the LZ77 + Huffman
tests.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 24 Nov 2022 09:51:01 +0000 (22:51 +1300)]
lib/compression: add test data for lzxpress plain compression
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Wed, 23 Nov 2022 22:11:15 +0000 (11:11 +1300)]
testdata: move compression examples to re-use with lzxpress plain
Everything that is in testdata/compression/lzxpress-huffman/ can also
be used for lzxpress plain tests, which is something we really need.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Wed, 23 Nov 2022 00:06:41 +0000 (13:06 +1300)]
lib/compression/lzx-plain: relax size requirements on long file
We are going to change from a slow exact match algorithm to a fast
heuristic search that will not always get the same results as the
exhaustive search.
To be precise, a million zeros will compress to 112 rather than 93 bytes.
We don't insist on an exact size, because that is not an issue here.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Tue, 22 Nov 2022 23:01:15 +0000 (12:01 +1300)]
lib/comression: convert test_lzxpress_plain to cmocka
Mainly so I can go
make bin/test_lzxpress_plain && bin/test_lzxpress_plain
valgrind bin/test_lzxpress_plain
rr bin/test_lzxpress_plain
rr replay
in a tight loop.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 23:45:12 +0000 (12:45 +1300)]
selftest: be less confident in commending st/summary
st/summary is useless. If you'll find anything, it'll be in st/subunit.
However, in case *something* useful ever ends up there we still mention it.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Sun, 20 Nov 2022 22:25:20 +0000 (11:25 +1300)]
lib/compression: add test scripts README
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Sun, 20 Nov 2022 21:17:54 +0000 (10:17 +1300)]
lib/compression: test util to generate fuzzing seeds
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Mon, 21 Nov 2022 19:49:39 +0000 (08:49 +1300)]
lib/compression: Windows utility to generate test vectors
If compiled on Windows using Cygwin, MSYS2, or similar, this will output
compressed versions of files exactly as specified by MZ-XCA, if the
following conditions are met:
1. The file > 300 bytes.
2. The compressed file is smaller than the decompressed file.
Otherwise it returns the data unchanged. Without warning; that's just
how the API works.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Mon, 21 Nov 2022 19:35:14 +0000 (08:35 +1300)]
lib/compression: script to test 3 byte hash
Compression uses a 3 byte hash remember LZ77 matches in a 14-bit table.
This script runs the hash over all 16M combinations, then again over
all ASCII combinations, counting collisions to find hot-spots.
If you think you have a better hash, you are probably right, but you
should try it here -- alter h() -- before committing to it. This one is
literally the first one I thought of.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Mon, 21 Nov 2022 19:23:30 +0000 (08:23 +1300)]
lib/compression: helper script to make unbalanced data
Huffman tree re-quantisation and perhaps other code paths are only
triggered by pathological data like this.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 07:02:21 +0000 (20:02 +1300)]
lib/compression: add a debug script to describe headers
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 03:09:39 +0000 (16:09 +1300)]
fuzz: add fuzz_lzxpress_huffman_round_trip
This compresses some data, decompresses it, and asserts that the
result is identical to the original string.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 03:09:26 +0000 (16:09 +1300)]
fuzz: add fuzz_lzxpress_huffman_compress
This differs from fuzz_lzxpress_huffman_round_trip (next commit) in
that the output buffer might be too small for the compressed data, in
which case we want to see an error and not a crash.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 03:08:52 +0000 (16:08 +1300)]
fuzz: add fuzz_lzxpress_huffman_decompress
Most strings will not successfully decompress, which is OK. What we
care about of course is memory safety.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 20:54:57 +0000 (09:54 +1300)]
lib/compression/tests: add lzhuffman timer functions
With LZXHUFF_DEBUG_VERBOSE set, we measure the compression and
decompression rate relative to the decompressed size.
On reasonably long strings on my laptop, compiled with -O0, it turns
out to between 20 and 500 MB/s, both ways, depending on the complexity
of the string. Very short strings are of course dominated by overhead.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Fri, 18 Nov 2022 02:54:37 +0000 (15:54 +1300)]
lib/compression: debug routines for lzxpress-huffman
If you need to see a Huffman tree (and sometimes you do), set
DEBUG_HUFFMAN_TREE to true at the top of lzxpress_huffman.c, and run:
make bin/test_lzx_huffman && bin/test_lzx_huffman
Actually, that will show you hundreds of trees, and you'll be glad of
that if you are ever trying to understand this.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Sun, 20 Nov 2022 21:23:53 +0000 (10:23 +1300)]
lib/compression/lzhuff: add debug flag to skip LZ77
Encoding without LZ77 matches is valid, and it is useful for isolating
bugs.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 10:14:58 +0000 (23:14 +1300)]
lib/compression: LZ77 + Huffman compression
This compresses files as described in MS-XCA 2.2, and as decompressed
by the decompressor in the previous commit.
As with the decompressor, there are two public functions -- one that
uses a talloc context, and one that uses pre-allocated memory. The
compressor requires a tightly bound amount of auxillary memory
(>220kB) in a few different buffers, which is all gathered together in
the public struct lzxhuff_compressor_mem. An instantiated but not
initialised copy of this struct is required by the non-talloc
function; it can be used over and over again.
Our compression speed is about the same as the decompression speed
(between 20 and 500 MB/s on this laptop, depending on the data), and
our compression ratio is very similar to that of Windows.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 01:24:52 +0000 (14:24 +1300)]
lib/compression: add LZ77 + Huffman decompression
This format is described in [MS-XCA] 2.1 and 2.2, with exegesis in
many posts on the cifs-protocol list[1].
The two public functions are:
ssize_t lzxpress_huffman_decompress(const uint8_t *input,
size_t input_size,
uint8_t *output,
size_t output_size);
uint8_t *lzxpress_huffman_decompress_talloc(TALLOC_CTX *mem_ctx,
const uint8_t *input_bytes,
size_t input_size,
size_t output_size);
In both cases the caller needs to know the *exact* decompressed size,
which is essential for decompression. The _talloc version allocates
the buffer for you, and uses the talloc context to allocate a 128k
working buffer. THe non-talloc function will allocate the working
buffer on the stack.
This compression format gives better compression for messages of
several kilobytes than the "plain" LXZPRESS compression, but is
probably a bit slower to decompress and is certainly worse for very
short messages, having a fixed 256 byte overhead for the first Huffman
table.
Experiments show decompression rates between 20 and 500 MB per second,
depending on the compression ratio and data size, on an i5-1135G7 with
no compiler optimisations.
This compression format is used in AD claims and in SMB, but that
doesn't happen with this commit.
I will not try to describe LZ77 or Huffman encoding here. Don't expect
an answer in MS-XCA either; instead read the code and/or Wikipedia.
[1] Much of that starts here:
https://lists.samba.org/archive/cifs-protocol/2022-October/
but there's more earlier, particularly in June/July 2020, when
Aurélien Aptel was working on an implementation that ended up in
Wireshark.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Pair-programmed-with: Joseph Sutton <josephsutton@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 03:07:08 +0000 (16:07 +1300)]
testdata: add test vectors for LZ77+Huffman [de-]compression
Some of the decompressed files were found via fuzzing, some are public
domain texts, and some are designed to test one aspect or another of
the format. For example, some aspects of Huffman tree creation can
only be tested when there is an extreme imbalance in the frequency of
symbols.
See the README for what files are where.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Tue, 22 Nov 2022 23:10:20 +0000 (12:10 +1300)]
test/source_chars: ignore testdata/compression
We are going to have all kinds of rubbish there.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 03:07:37 +0000 (16:07 +1300)]
lib/compression: move lzxpress_plain test into tests/
We are going to add more tests for lib/compression, and they can't all
be called "testsuite.c".
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Wed, 30 Nov 2022 03:59:51 +0000 (16:59 +1300)]
fuzz: add fuzzers for stable_sort
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Wed, 28 Sep 2022 01:40:10 +0000 (14:40 +1300)]
util: add stable sort functions
Sometimes (e.g. in lzxpress Huffman encoding, and in some of our
tests: c.f. https://lists.samba.org/archive/samba-technical/2018-March/126010.html)
we want a stable sort algorithm (meaning one that retains the previous
order of items that compare equal).
The GNU libc qsort() is *usually* stable, in that it first tries to
use a mergesort but reverts to quicksort if the necessary allocations
fail. That has led Samba developers to unthinkingly assume qsort() is
stable which is not the case on many platforms, and might not always
be on GNU/Linuxes either.
This adds four functions. stable_sort() sorts an array, and requires
an auxiliary working array of the same size. stable_sort_talloc()
takes a talloc context so it ca create a working array and call
stable_sort(). stable_sort_r() takes an opaque context blob that gets
passed to the compare function, like qsort_r() and ldb_qsort(). And
stable_sort_talloc_r() rounds out the quadrant.
These are LGPL so that the can be used in ldb, which has problems with
unstable sort.
The tests are borrowed and extended from test_ldb_qsort.c.
When sorting non-trivial structs this is roughly as fast as GNU qsort,
but GNU qsort has optimisations for small items, using direct
assignments of rather than memcpy where the size allows the item to be
cast as some kind of int.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Jeremy Allison [Fri, 18 Nov 2022 18:50:35 +0000 (10:50 -0800)]
s3: smbd: Fix schedule_smb2_aio_read() to allow the last read in a compound to go async.
Remove knownfail.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Thu Dec 1 16:04:07 UTC 2022 on sn-devel-184
Jeremy Allison [Fri, 18 Nov 2022 18:45:19 +0000 (10:45 -0800)]
s3: smbd: Fix schedule_aio_smb2_write() to allow the last write in a compound to go async.
Remove knownfail.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Jeremy Allison [Thu, 17 Nov 2022 23:50:30 +0000 (15:50 -0800)]
s4: torture: Add compound_async.read_read test to show we don't go async on the last read in a compound.
Add knownfail.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Jeremy Allison [Thu, 17 Nov 2022 23:39:16 +0000 (15:39 -0800)]
s4: torture: Add compound_async.write_write test to show we don't go async on the last write in a compound.
Add knownfail.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Jeremy Allison [Fri, 18 Nov 2022 21:30:05 +0000 (13:30 -0800)]
s4: torture: Tweak the compound padding streamfile test to send 3 reads instead of 2, and check the middle read padding.
The protocol allows the last read in a related compound to be split
off and possibly go async (and smbd soon will do this). If the
last read is split off, then the padding is different. By sending
3 reads and checking the padding on the 2nd read, we cope with
the smbd change and are still correctly checking the padding
on a compound related read.
Do this for the stream filename compound padding test.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Jeremy Allison [Fri, 18 Nov 2022 21:23:48 +0000 (13:23 -0800)]
s4: torture: Tweak the compound padding basefile test to send 3 reads instead of 2, and check the middle read padding.
The protocol allows the last read in a related compound to be split
off and possibly go async (and smbd soon will do this). If the
last read is split off, then the padding is different. By sending
3 reads and checking the padding on the 2nd read, we cope with
the smbd change and are still correctly checking the padding
on a compound related read.
Do this for the base filename compound padding test.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Jeremy Allison [Fri, 18 Nov 2022 17:53:23 +0000 (09:53 -0800)]
s3: tests: Change smb2.compound_async to run against share aio_delay_inject instead of tmp.
It doesn't hurt the fsync compound async tests, and we need this for
the next commits to ensure smb2_read/smb2_write compound tests take
longer than 500ms so can be sure the last read/write in the compound
will go async.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Andreas Schneider [Wed, 30 Nov 2022 17:23:17 +0000 (18:23 +0100)]
s4:torture: Fix segfault in multichannel test
The timer for the timeout_cb() handler was created on a memory context
which doesn't get freed, so the timer was still valid when running
the next test and fired there. It was then writing into random memory
leading to segfaults.
Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Dec 1 15:03:19 UTC 2022 on sn-devel-184
Volker Lendecke [Tue, 22 Nov 2022 15:00:53 +0000 (16:00 +0100)]
smbd: Allow POSIX getinfo levels for smb3 unix extensions
Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: David Mulder <dmulder@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Tue Nov 29 11:23:58 UTC 2022 on sn-devel-184
David Mulder [Thu, 3 Nov 2022 16:28:58 +0000 (10:28 -0600)]
s3: Test that store_smb2_posix_info hides info for '..'
Signed-off-by: David Mulder <dmulder@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
David Mulder [Tue, 18 Oct 2022 13:37:47 +0000 (07:37 -0600)]
s3: smbd: store_smb2_posix_info hide info for '..'
When receiving a query for '..', hide the owner
and group sids, the inode, and the dev id.
Signed-off-by: David Mulder <dmulder@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
David Mulder [Mon, 12 Sep 2022 23:09:50 +0000 (16:09 -0700)]
smbd: Implement SMB2_FS_POSIX_INFORMATION_INTERNAL
Signed-off-by: David Mulder <dmulder@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
David Mulder [Fri, 5 Aug 2022 20:00:30 +0000 (14:00 -0600)]
tests/s3: Test file/dir permissions with SMB3 posix
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
David Mulder [Fri, 5 Aug 2022 19:05:48 +0000 (13:05 -0600)]
tests/s3: Test case sensitive open with SMB3 posix
Disabled because we don't handle posix paths
correctly yet.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
David Mulder [Fri, 5 Aug 2022 16:56:32 +0000 (10:56 -0600)]
tests/s3: Test delete on close with SMB3 posix
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
David Mulder [Fri, 8 Jul 2022 19:15:51 +0000 (13:15 -0600)]
libcli: Add client support for SMB2_FILE_POSIX_INFORMATION
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
David Mulder [Thu, 7 Jul 2022 18:57:01 +0000 (12:57 -0600)]
tests/s3: Test reserved chars in posix filename
Disabled because we don't handle posix paths
correctly yet.
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
David Mulder [Fri, 17 Jun 2022 21:06:29 +0000 (15:06 -0600)]
smbd: Implement SMB2_FILE_POSIX_INFORMATION in smbd_marshall_dir_entry
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
David Mulder [Wed, 15 Jun 2022 21:39:00 +0000 (15:39 -0600)]
tests/s3: Test SMB2_FIND_POSIX_INFORMATION dir query
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
David Mulder [Tue, 20 Sep 2022 16:28:20 +0000 (10:28 -0600)]
libsmb: Allow listing with posix context
Signed-off-by: David Mulder <dmulder@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
David Mulder [Wed, 15 Jun 2022 19:20:30 +0000 (13:20 -0600)]
libsmb: Make info_level configurable in dir listing
This was hard coded to SMB2_FIND_ID_BOTH_DIRECTORY_INFO
Signed-off-by: David Mulder <dmulder@suse.com>
Reviewed-by: Volker Lendecke <vl@samba.org>
Jeremy Allison [Mon, 18 Dec 2017 21:27:06 +0000 (13:27 -0800)]
smbd: Plumb SMB2_FIND_POSIX_INFORMATION through the directory reading code.
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Jeremy Allison [Thu, 14 Dec 2017 23:18:21 +0000 (15:18 -0800)]
s3: smbd: Add SMB2_FILE_POSIX_INFORMATION getinfo info level (100 on the wire).
Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Ralph Boehme [Tue, 22 Nov 2022 06:31:52 +0000 (07:31 +0100)]
smbd: reject FILE_ATTRIBUTE_TEMPORARY on directories
Cf MS-FSA 2.1.5.14.2
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15252
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Mon Nov 28 10:14:12 UTC 2022 on sn-devel-184
Ralph Boehme [Tue, 22 Nov 2022 09:45:35 +0000 (10:45 +0100)]
torture: add a test trying to set FILE_ATTRIBUTE_TEMPORARY on a directory
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15252
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Stefan Metzmacher [Wed, 27 Mar 2019 03:34:12 +0000 (04:34 +0100)]
vfs: fix the build of nfs4acl_xattr_ without rpc/xdr.h support
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Fri Nov 25 06:07:32 UTC 2022 on sn-devel-184
Ralph Boehme [Tue, 10 Jan 2017 11:22:28 +0000 (12:22 +0100)]
net: use correct printf format, fi3_id is an uint32_t
Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Thu Nov 24 16:39:12 UTC 2022 on sn-devel-184
Stefan Metzmacher [Tue, 22 Nov 2022 09:41:39 +0000 (10:41 +0100)]
gitlab-ci: do some basic testing on ubuntu1804-32bit
For now we allow build warnings and only do some basic testing.
We also ignore timestamp related problems, as well as some charset
failures.
Over time we should try to address the situation by not allowing warnings
and verify if expected failures are harmless or not.
But it's already much better then having no 32bit testing at all!
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Nov 24 12:05:26 UTC 2022 on sn-devel-184
Stefan Metzmacher [Tue, 22 Nov 2022 09:31:19 +0000 (10:31 +0100)]
selftest: add --default-ldb-backend option
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Stefan Metzmacher [Fri, 4 Nov 2022 10:23:07 +0000 (10:23 +0000)]
selftest: samba-ktest-mit also needs $ENV{KRB5RCACHETYPE} = "none"
We need to pass --mitkrb5 to selftest.pl in all cases we use
system mit kerberos not only when we also test the kdc.
We can't use a replay cache in selftest verifies the stat.st_uid
against getuid().
BTW: while debugging this on ubuntu 22.04 I exported
KRB5_TRACE="/dev/stderr", which means we get tracing into
the servers log file and into selftest_prefix/subunit for the client...
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Stefan Metzmacher [Wed, 16 Nov 2022 11:08:45 +0000 (12:08 +0100)]
CVE-2022-42898: HEIMDAL: lib/krb5: fix _krb5_get_int64 on systems where 'unsigned long' is just 32-bit
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15203
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Stefan Metzmacher [Wed, 23 Nov 2022 11:14:12 +0000 (12:14 +0100)]
third_party: Update socket_wrapper to version 1.3.5
This injects O_LARGEFILE as needed.
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Stefan Metzmacher [Wed, 23 Nov 2022 10:38:20 +0000 (11:38 +0100)]
lib/replace: let rep_openat2() inject O_LARGEFILE as needed
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15251
Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>