lib/pthreadpool: fix a memory leak

When copying large files from the server to the client with aio enabled
we noticed that smbd kept growing RSS and VSZ.

valgrind was reporting:

==2503== 4,093,440 bytes in 6,560 blocks are possibly lost in loss record 460 of 460
==2503==    at 0x4C299CE: calloc (vg_replace_malloc.c:711)
==2503==    by 0x4011C24: _dl_allocate_tls (in /usr/lib64/ld-2.17.so)
==2503==    by 0x4E3C960: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.17.so)
==2503==    by 0x9B298AE: pthreadpool_add_job (in /usr/lib64/samba/libmessages-dgm-samba4.so)
==2503==    by 0x9B29FDC: pthreadpool_tevent_job_send (in /usr/lib64/samba/libmessages-dgm-samba4.so)
==2503==    by 0x56A78EF: ??? (in /usr/lib64/samba/libsmbd-base-samba4.so)
==2503==    by 0x55D86B7: smb_vfs_call_pread_send (in /usr/lib64/samba/libsmbd-base-samba4.so)
==2503==    by 0x55F7543: schedule_smb2_aio_read (in /usr/lib64/samba/libsmbd-base-samba4.so)
==2503==    by 0x5608F57: smbd_smb2_request_process_read (in /usr/lib64/samba/libsmbd-base-samba4.so)
==2503==    by 0x55FCB6C: smbd_smb2_request_dispatch (in /usr/lib64/samba/libsmbd-base-samba4.so)
==2503==    by 0x55FD7DC: ??? (in /usr/lib64/samba/libsmbd-base-samba4.so)
==2503==    by 0x641B977: ??? (in /usr/lib64/samba/libtevent.so.0.9.31)

The problem seems to be caused by worked threads that are not properly
started in detached state and thus their tls is not reclaimed upon
thread termination.

In pthreadpool.c we prepare a pthread attribute with
PTHREAD_CREATE_DETACHED, but we don't pass it to pthread_create().

Bug: https://bugzilla.samba.org/show_bug.cgi?id=12624

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Fri Mar 10 22:06:02 CET 2017 on sn-devel-144

objectclass_attrs: Remove schema copy shallow from attr_handler2

This appears quite expensive (particularly in provision), and also
unnecessary.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Fri Mar 10 15:34:39 CET 2017 on sn-devel-144

s4:kdc: disable principal based autodetected referral detection

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

HEIMDAL:kdc: make it possible to disable the principal based referral detection

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>

s3:gse: Correctly handle external trusts with MIT

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

s3:gse: Check if we have a target_princpal set we should use

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

s3:gse: Move setup of service_principal to update function

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

s3:gse: Pass down the gensec_security pointer

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

krb5_wrap: Remove obsolete smb_krb5_get_principal_from_service_hostname()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

s3:gse: Use smb_krb5_get_realm_from_hostname()

With credentials for administrator@FOREST1.EXAMPLE.COM
this patch changes the target_principal for
the ldap service of host dc2.forest2.example.com
from

  ldap/dc2.forest2.example.com@FOREST1.EXAMPLE.COM

to

  ldap/dc2.forest2.example.com@FOREST2.EXAMPLE.COM

Typically ldap/dc2.forest2.example.com@FOREST1.EXAMPLE.COM
should be used in order to allow the KDC of FOREST1.EXAMPLE.COM
to generate a referral ticket for
krbtgt/FOREST2.EXAMPLE.COM@FOREST1.EXAMPLE.COM.

The problem is that KDCs only return such referral tickets
if there's a forest trust between FOREST1.EXAMPLE.COM
and FOREST2.EXAMPLE.COM. If there's only an external domain
trust between FOREST1.EXAMPLE.COM and FOREST2.EXAMPLE.COM
the KDC of FOREST1.EXAMPLE.COM will respond with S_PRINCIPAL_UNKNOWN
when being asked for ldap/dc2.forest2.example.com@FOREST1.EXAMPLE.COM.

In the case of an external trust the client can still ask
explicitly for krbtgt/FOREST2.EXAMPLE.COM@FOREST1.EXAMPLE.COM
and the KDC of FOREST1.EXAMPLE.COM will generate it.

From there the client can use the
krbtgt/FOREST2.EXAMPLE.COM@FOREST1.EXAMPLE.COM
ticket and ask a KDC of FOREST2.EXAMPLE.COM for a
service ticket for ldap/dc2.forest2.example.com@FOREST2.EXAMPLE.COM.

With Heimdal we'll get the fallback on S_PRINCIPAL_UNKNOWN behavior
when we pass ldap/dc2.forest2.example.com@FOREST2.EXAMPLE.COM as
target principal. As _krb5_get_cred_kdc_any() first calls
get_cred_kdc_referral() (which always starts with the client realm)
and falls back to get_cred_kdc_capath() (which starts with the given realm).

MIT krb5 only tries the given realm of the target principal,
if we want to autodetect support for transitive forest trusts,
we'll have to do the fallback ourself.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

s4:gensec_gssapi: Correctly handle external trusts with MIT

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

s4:gensec_gssapi: Use smb_krb5_get_realm_from_hostname()

With credentials for administrator@FOREST1.EXAMPLE.COM
this patch changes the target_principal for
the ldap service of host dc2.forest2.example.com
from

  ldap/dc2.forest2.example.com@FOREST1.EXAMPLE.COM

to

  ldap/dc2.forest2.example.com@FOREST2.EXAMPLE.COM

Typically ldap/dc2.forest2.example.com@FOREST1.EXAMPLE.COM
should be used in order to allow the KDC of FOREST1.EXAMPLE.COM
to generate a referral ticket for
krbtgt/FOREST2.EXAMPLE.COM@FOREST1.EXAMPLE.COM.

The problem is that KDCs only return such referral tickets
if there's a forest trust between FOREST1.EXAMPLE.COM
and FOREST2.EXAMPLE.COM. If there's only an external domain
trust between FOREST1.EXAMPLE.COM and FOREST2.EXAMPLE.COM
the KDC of FOREST1.EXAMPLE.COM will respond with S_PRINCIPAL_UNKNOWN
when being asked for ldap/dc2.forest2.example.com@FOREST1.EXAMPLE.COM.

In the case of an external trust the client can still ask
explicitly for krbtgt/FOREST2.EXAMPLE.COM@FOREST1.EXAMPLE.COM
and the KDC of FOREST1.EXAMPLE.COM will generate it.

From there the client can use the
krbtgt/FOREST2.EXAMPLE.COM@FOREST1.EXAMPLE.COM
ticket and ask a KDC of FOREST2.EXAMPLE.COM for a
service ticket for ldap/dc2.forest2.example.com@FOREST2.EXAMPLE.COM.

With Heimdal we'll get the fallback on S_PRINCIPAL_UNKNOWN behavior
when we pass ldap/dc2.forest2.example.com@FOREST2.EXAMPLE.COM as
target principal. As _krb5_get_cred_kdc_any() first calls
get_cred_kdc_referral() (which always starts with the client realm)
and falls back to get_cred_kdc_capath() (which starts with the given realm).

MIT krb5 only tries the given realm of the target principal,
if we want to autodetect support for transitive forest trusts,
we'll have to do the fallback ourself.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

s4:gensec_gssapi: Move setup of service_principal to update function

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

s4:gensec-gssapi: Create a helper function to setup server_principal

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

krb5_wrap: Make smb_krb5_get_realm_from_hostname() public

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

krb5_wrap: pass client_realm to smb_krb5_get_realm_from_hostname()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

krb5_wrap: Try to guess the correct realm from the service hostname

If we do not get a realm mapping from the krb5.conf or from the Kerberos
library try to guess it from the service hostname. The guessing of the
realm from the service hostname is already implemented in Heimdal. This
makes the behavior of smb_krb5_get_realm_from_hostname() consistent
with both MIT and Heimdal.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

krb5_wrap: Do not return an empty realm from smb_krb5_get_realm_from_hostname()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>

Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Stefan Metzmacher <metze@samba.org>

testprogs: Add kinit_trusts tests with smbclient4

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

testprogs: Use smbclient by default in test_kinit_trusts

This is the tool we use by default and we should test with it.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=12554

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

waf: disable-python - don't include python.h in test_headers.c

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Mar 10 11:24:13 CET 2017 on sn-devel-144

autobuild: Add nopython environment to test --disable-python builds (but without tests)

This ensures we keep this option building as we extend our use of python.

The rule is that new features and changes to existing features that
require python are most welcome, they just need to be disabled for the
minimalistic targets we still ecourage Samba on, that typically just
want smbd

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - don't build torture bits

samba-net being disabled causes a chain of dependency or proto.h-based
missing code issues that require a number of modules or subsystems
to be disabled in samba4/torture.

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - don't build samba-policy

samba-policy requires samba-net which requires PROVISION, which
is disabled when python isn't available.

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - don't build samba-net

samba-net requires PROVISION, which is disabled when python isn't available.

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - don't build pyrpc_util, dcerpc.py

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - don't build PROVISION, pyparam_util

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - don't build python/

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - align tdb's wscript

Drop the configure option for --disable-python as it is now
global in wafsamba.

If samba is set to use a system copy of tdb, and tdb wasn't built
with python support, then the system pytevent will not be found.  If
samba is being built without python support then pytdb is not needed,
so do not bother to try and find it.

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - align tevent wscript

Drop the configure option for --disable-python as it is now
global in wafsamba.

If samba is set to use a system copy of tevent, and tevent wasn't built
with python support, then the system pytevent will not be found.  If
samba is being built without python support then pytevent is not needed,
so do not bother to try and find it.

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - align ldb's wscript

If samba is set to use a system copy of ldb, and ldb wasn't built with
python support, then no system pyldb-util will be found.  If samba is
being built without python support then pyldb-util isn not needed, so
do not bother to try and find it.

The system ldb check had to be duplicated due to the earlier commits
which changed order of ldb and pyldb-util checks, and by association
also added a dependency of pyldb-util onto ldb.  This seemed cleaner
than messing with variables.

The build configuration for pyldb-util needs to exist even if it's
not being built, so that dependency resolution can occur throughout
the rest of the samba build system -- this required dropping the higher
level conditional and using the enabled= parameter instead.

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - align talloc's wscript

Drop the configure option for --disable-python as it is now
global in wafsamba

If samba is set to use a system copy of talloc, and talloc wasn't built
with python support, then the system pytalloc-util will not be found.
If samba is being built without python support then pytalloc-util is not
needed, so do not bother to try and find it.

The build configuration for pytalloc-util needs to exist even if it's
not being built, so that dependency resolution can occur throughout
the rest of the samba build system -- this required dropping the higher
level conditional and using the enabled= parameter instead.

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - configuration adjustments

Adjust configuration to accomodate when --disable-python is set:

- Error when AD-DC is still enabled (and others later as needed)

- Set mandatory=false on SAMBA_CHECK_PYTHON_HEADERS

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - add option globally to build system

This commit adds --disable-python as an option to the build system.
It adds PYTHON_BUILD_IS_ENABLED() to bld, to be used with enabled=
on other modules, and adjusts SAMBA_PYTHON() to set enabled=False
if PYTHON_BUILD_IS_ENABLED() is false.

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

waf: disable-python - fix ctdb configuration

When ctdb is built in standalone mode, it turned off the python
requirement for submodules by setting Options.options.disable_python
to True before checking for its own (non-optional) python support.

Ad ctdb does not need python for itself or any of the submodules
it is built against, the safest solution seems to be to allow
the python and python-headers checks to not find python.

Signed-off-by: Ian Stakenvicius <axs@gentoo.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: Port the samba.net module to Python 3

Signed-off-by: Petr Viktorin <pviktori@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: Remove unused import PY3

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: selftest: Add possibility to run old Python test suites with Python 3

Add possibility to execute old Python test suites with Python 3
and enable tests with Python 3 of ported samba.gensec module.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.gensec: Port module to Python 3 compatible form

Port samba.gensec and samba.tests.gensec modules to Python 3
compatible form, enable execution of tests with Python 3 and
remove unused import of samba.gensec from samba.tests module
__init__.py file.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.gensec: Fix error handling in set_credentials() function

Add `return NULL;` to error handling part of `set_credentials()`
function.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: selftests: Enable samba.getopt tests execution with Python 3

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.getopt: Port module to Python 3 compatible form

Port samba.getopt module to Python 3 compatible form.

Remove unused and untested `get_hostconfig()` function. Andrew Bartlett
suggested this removal because it is the simpliest way how to break
a long dependency line of Python modules which have to be ported
at once.
More info: https://lists.samba.org/archive/samba-technical/2017-January/118150.html

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.tests.core: Port and enable core tests in Python 3

Port samba core tests to Python 3 compatible form and enable their
execution with Python 3.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.tests: Move import of ported modules out of PY3 condition

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba._ldb: Port of samba._ldb to Python 3 compatible form

Port of samba._ldb Python module to Python 3 compatible form.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.tests.auth: Add tests for samba.auth module

Add some tests which test that `system_session` object has
correct attributes and methods.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.auth: Port samba.auth to Python 3 compatible form

Port samba.auth Python module to Python 3 compatible form and
enable tests execution with Python 3.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: wscript_build: Build some DCE/RPC modules with Python 3

Samba.auth Python module depends on a lot of DCE/RPC modules which
have to be built with Python 3 to make port of samba.auth to
Python 3 possible.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.dcerpc: Port security module to Python 3 comp. form

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dcerpc/misc tests: asset GUID ordering in python 2 and 3

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

python: samba.tests.dcerpc.misc: Port and enable tests

Port tests of samba.dcerpc.misc module to Python 3 compatible form
and enable their execution with Python 3.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.dcerpc: Port RPC related stuff to Python 3

Port RPC related stuff like samba.dcerpc.misc and samba.dcerpc
Python modules and pyrpc_util to Python 3 compatible form.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Pair-programmed-by: Andrew Bartlett <abartlet@samba.org>

python: pidl: Port Python interface generator

Port PIDL generator of Python interfaces to generate interfaces in
Python 3 compatible form.

Python 2.7 is now required, so we can use PyCapsule in both versions.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Pair-programmed-by: Andrew Bartlett <abartlet@samba.org>

python: samba.tests: Enable Python 3 tests for ported modules

Enable tests with Python 3 for Python 3 compatible modules.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

buildtools: Work around a . being in the target name when building python3 helpers

The pyparam_util module becomes pyparam_util.cpython_35m_x86_64_linux_gnu but
the command line parser for -D stops at the first .

That we even set -DSTATIC_subsystem_MODULES_PROTO for these subsystems without
any modules ever declared is left for another time

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: wscript_build: Build some modules for Python 3

Update a few wscript_build files to build Python 3-compatible modules
for Python 3.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: Make top-level samba modules Python 3 compatible

New file compat.py will help with porting to Python 3. For now, it
contains only PY3 variable based on six.PY3 which simplifies
condition mentioned below.

The added `if not PY3` conditions enable us to bootstrap running
tests with Python 3 even if most modules are not ported yet.
The plan is to move modules outside this condition as they are ported.
The `PY3` condition is currently used only in tests and for
the samba._ldb module which is not ported yet and has a lot of
dependencies.

The other changes are related to differences between Python 2 and 3.
Python 2.6 introduced the `0o` prefix for octal literals as an
alternative to plain `0`. In Python 3, support for plain `0` is
dropped and octal literals have to start with `0o` prefix.
Python 2.6 introduced a clearer `except` syntax:
`except ExceptionType as target:` instead of
`except ExceptionType, target:`. In Python 3, the old syntax
is no longer allowed.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.tests.dcerpc: Move Class RawDCERPCTest to separated file.

The class is quite big, used in only one place, and it complicates
situation around bootstrapping of Python 3 port.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.tests.glue: Add new tests for samba._glue.

Add new file with tests of samba._glue module.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba._glue: Port samba._glue module to Python 3.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.tests.param: Add missing tests

Add some new tests of samba.param Python bindings.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.param: Port param module to Python 3

Port Python bindings of samba.param module to
Python3-compatible form.

Because native Python file objects are officially
no longer backed by FILE*, API of some _dump()
functions is changed. File argument is now
optional and contains only name of file. Stdout
is default if no file name is specified. Otherwise
opening and closing files is done on C layer
instead of Python.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.tests.credentials: Python 3 compatible tests

Port test of pycredentials to Python 3 compatible form.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python: samba.credentials: Port pycredentials.c to Python3-compatible form.

Port Python bindings of samba.credentials module to
Python3-compatible form using macros from py3compat.h.

Signed-off-by: Lumir Balhar <lbalhar@redhat.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

lib/ldb: Enable use of a python3 pyldb-util system library

To do this, we have to install a .pc file for the python3 pyldb-util

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Incorportaing fixes by Petr Viktorin <pviktori@redhat.com>

Signed-off-by: Petr Viktorin <pviktori@redhat.com>

talloc: use the system pytalloc-util for python3 as well

This involves installing a .pc file for the python3 library as well

To get the .pc file generated and installed is quite a mission, we
have to rework the talloc build system to ensure that the second 'env'
created for EXTRA_PYTHON has everything set up on it, the
TALLOC_VERSION in particular.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Incorportaing fixes by Petr Viktorin <pviktori@redhat.com>

Signed-off-by: Petr Viktorin <pviktori@redhat.com>

scripts/traffic_summary: documentation typo

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

./examples/scripts/SambaConfig.py: fix typo in "continue"

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

python/examples/winreg: two variable name typos on a single line

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

python sites/subnets: correctly spell variable name

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

python provision: FDSBackend takes forced uri

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

python/remove_dc: avoid using non-existent variable

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet@samba.org>

samba-tool domain: correctly spell variable name

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

python/join: correct spelling of "ctx.del_noerror"

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

selftest: remove unused broken client.py

Nothing uses this, and pyflakes points out it is unusable:

./selftest/client.py:60: undefined name 'prefix_abs'
./selftest/client.py:69: undefined name 'opts'
./selftest/client.py:70: undefined name 'interfaces'

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gitignore: add some hidden files

.gdb_history is generated by gdb,
.emacs* are generated by emacs, and
.clang* by clang.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

perftests/ad_dc_search: do less work in expensive member searches

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

pyldb: p3k readiness: allow single unicode string in msg element

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

talloc: fix doxygen of talloc_move

talloc_move cannot fail.

Signed-off-by: Uri Simchoni <uri@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Fri Mar 10 07:30:40 CET 2017 on sn-devel-144

auth_ntdomain3: Correctly handle !authoritative

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

auth_winbind4: Correctly handle !authoritative

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

auth_winbind3: Correctly handle !authoritative

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

s3/smbd: add my copyright to open.c

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s4/torture: some tests for kernel oplocks

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3/selftest: adopt config.h check from source4

No change in behaviour.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3/smbd: fix deferred open with streams and kernel oplocks

I noticed smbd can get stuck in an open() call with kernel oplocks
enabled and named streams (provided by vfs_streams_xattr):

- client opens a file and with an exclusive oplock

- client starts writing to the file

- client opens an existing stream of the file

- the smbd process gets stuck in an open()

What happens is:

we had setup a locking.tdb record watch in defer_open(), the watch was
triggered, we reattempted the open and got stuck in a blocking open
because the oplock holder (ourselves) hadn't given up the oplock yet.

Cf e576bf5310bc9de9686a71539e9a1b60b4fba5cc for the commit that added
the kernel oplock retry logic. tldr: with kernel oplocks the first open
is non-blocking, but the second one is blocking.

Detailed analysis follows.

When opening a named stream of a file, Samba internally opens the
underlying "base" file first. This internal open of the basefile suceeds
and does *not* trigger an oplock break (because it is an internal open
that doesn't call open() at all) but it is added as an entry to the
locking.tdb record of the file.

Next, the stream open ends up in streams_xattr where a non-blocking
open() on the base file is called. This open fails with EWOULDBLOCK
because we have another fd with a kernel oplock on the file.

So we call defer_open() which sets up a watch on the locking.tdb record.

In the subsequent error unwinding code in open_file_ntcreate() and
callers we close the internal open file handle of the basefile which
also removes the entry from the locking.tdb record and so *changes the
record*.

This fires the record watch and in the callback defer_open_done() we
don't check whether the condition (oplock gone) we're interested in is
actually met. The callback blindly reschedules the open request with
schedule_deferred_open_message_smb().

schedule_deferred_open_message_smb() schedules an immediate tevent event
which has precedence over the IPC fd events in messaging, so the open is
always (!) reattempted before processing the oplock break message.

As explained above, this second open will be a blocking one so we get
stuck in a blocking open.

It doesn't help to make all opens non-blocking, that would just result
in a busy loop failing the open, as we never process the oplock break
message (remember, schedule_deferred_open_message_smb() used immediate
tevent events).

To fix this we must add some logic to the record watch callback to check
whether the record watch was done for a kernel oplock file and if yes,
check if the oplock state changed. If not, simply reschedule the
deferred open and keep waiting.

This logic is only needed for kernel oplocks, not for Samba-level
oplocks, because there's no risk of deadlocking, the worst that can
happen is a rescheduled open that fails again in the oplock checks and
gets deferred again.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3/smbd: all callers of defer_open() pass a lck

No change in behaviour. Update the function comment explaining how it
works and relies on lck for a record watch.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3/smbd: remove async_open arg from defer_open()

All remaining callers pass false.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3/smbd: fix schedule_async_open() timer

schedule_async_open() was calling defer_open with sharemode lock = NULL,
as a result there was never an active 20 s timeout.

This has been broken since the commits in

$ git log --reverse -p -10 8283fd0e0090ed12b0b12d5acb550642d621b026

Just roll our own deferred record instead of calling defer_open() and
also set up timer that, as a last resort, catches stuck opens and just
exits for now.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3/smbd: add and use retry_open() instead of defer_open() in two places

Add a new function that does an immediate open rescheduling.

The first deferred open this commit changes was never scheduled, as the
scheduling relies on a timeout of the watch on the sharemode lock.

This has been broken since the commits in

$ git log --reverse -p -10 8283fd0e0090ed12b0b12d5acb550642d621b026

That patchset added the dbwrap watch record logic to defer_open() and
removed the timers.

I'm doing this mainly to untangle the defer_open() logic which is
complicated by the lck arg.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3/smbd: simplify defer_open()

Add a helper function deferred_open_record_create() that creates a
deferred_open_record and let all callers pass all needed arguments
individually.

While we're at it, enhance the debug message in defer_open() to print
all variables.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3/smbd: req is already validated at the beginning of open_file_ntcreate()

req can't be NULL because the if condition surrounding this code checks
!(oplock_request & INTERNAL_OPEN_ONLY).

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3/smbd: add comments and some reformatting to open_file_ntcreate()

No change in behaviour.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3/smbd: add const to get_lease_type() args

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3/wscript: fix Linux kernel oplock detection

Fix a copy/paste error, the Linux kernel oplocks check was copied from
the change notify support check.

Bug: https://bugzilla.samba.org/show_bug.cgi?id=7537

Signed-off-by: Ralph Boehme <slow@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

winbindd: Remove an unused #define

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Uri Simchoni <uri@samba.org>
Autobuild-User(master): Uri Simchoni <uri@samba.org>
Autobuild-Date(master): Fri Mar 10 00:00:15 CET 2017 on sn-devel-144

winbind: Use talloc_strdup_upper where appropriate

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Uri Simchoni <uri@samba.org>

ldap_server: Fix a typo

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Uri Simchoni <uri@samba.org>

winbind: Fix a typo

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Uri Simchoni <uri@samba.org>

ldb: add LDB_FLG_DONT_CREATE_DB

This avoids creating an new tdb files on ldbsearch
or other callers which use LDB_FLG_DONT_CREATE_DB.

Signed-off-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Mar  9 16:02:21 CET 2017 on sn-devel-144