%%<[sambabg] ==== ====[plain] <[center] \Large '''Multi-Channel''' [center]> %%[frame]> %%[sambabg]> ==== Multi-Channel - General ==== <[block]{multiple transport connections in one SMB(3) session} * '''channel''': transport connection bound to a session * client decides which connections to bind and to use * session is valid as long as at least one channel is intact [block]> <[block]{two purposes} # increase throughput: #* use multiple connections of same type # improve fault tolerance: #* channel failure: replay/retry detection %%#* session is valid as long as one channel is still intact [block]> %% ==== Multi-Channel - General ==== %% %% <[block]{use case: channels of different type/quality} %% * use only the channels of best quality %% * fall back to inferior channels if superior ones fail %% * e.g.: laptop switching between WiFi and LAN (?) %% [block]> ==== Multi-Channel - Windows/Protocol ==== # establish initial session on TCP connection # find interfaces with interface discovery: \\ % @FSCTL\_QUERY\_NETWORK\_INTERFACE\_INFO@ # bind additional TCP (or later RDMA) connection (channel) to established SMB3 session (''session bind'') # Windows: uses connections of same (and best) quality # Windows: binds only to a single node # replay / retry mechanisms, epoch numbers ==== Multi-Channel $\in$ Samba ==== <[block]{samba/smbd: multi-process} * '''Originally:''' process $\Leftrightarrow$ TCP connection *<0> Want to avoid synchronization between smbds for disk access. *<0> '''Idea:''' transfer new TCP connection to existing smbd *<0> '''How?''' ==> use fd-passing (sendmsg/recvmsg) *<0> '''When?''' ** ''Natural choice'': at SessionSetup (Bind) ** !Samba's choice!: at Negotiate, based on ClientGUID [block]> ==== Multi-Channel $\in$ Samba ==== <[center] <<>> [center]> ==== Multi-Channel $\in$ Samba ==== <[block]{samba/smbd: multi-process} * '''Originally:''' process $\Leftrightarrow$ TCP connection * Want to avoid synchronization between smbds for disk access. *<2-> '''Idea:''' transfer new TCP connection to existing smbd *<0> '''How?''' ==> use fd-passing (sendmsg/recvmsg) *<0> '''When?''' ** ''Natural choice'': at SessionSetup (Bind) ** !Samba's choice!: at Negotiate, based on ClientGUID [block]> ==== Multi-Channel $\in$ Samba ==== -<1>{ <[center] <<>> [center]> } -<2>{ <[center] <<>> [center]> } -<3>{ <[center] <<>> [center]> } ==== Multi-Channel $\in$ Samba ==== <[block]{samba/smbd: multi-process} * '''Originally:''' process $\Leftrightarrow$ TCP connection * Want to avoid synchronization between smbds for disk access. * '''Idea:''' transfer new TCP connection to existing smbd * '''How?''' ==> use fd-passing (sendmsg/recvmsg) *<2-> '''When?''' ** ''Protocol choice'': at SessionSetup (Bind) ** !Samba's choice!: at Negotiate, based on ClientGUID [block]> ==== Multi-Channel $\in$ Samba : pass by ClientGUID ==== <[center] %%<<>> <<>> [center]> ==== Multi-Channel $\in$ Samba : pass by ClientGUID ==== <[block]{Wait a minute - what about performance?} * Single process... * But we use short-lived worker-pthreads for I/O ops! * Extensive benchmarks and tunings still to be done. * First benchmarks show $\ge$ 50\% increase going from 1 channel to 2 [block]> ==== Multi-Channel $\in$ Samba : Status ==== #<2-> messaging rewrite using unix dgm sockets with sendmsg [DONE,4.2] #<2-> add fd-passing to messaging [DONE,4.2] #<2-> preparations in internal structures [DONE,4.4] #<2-> prepare code to cope with multiple channels [DONE,4.4] #<2-> implement smbd message to pass a tcp socket [DONE,4.4] #<2-> transfer connection in Negotiate (by ClientGUID) [DONE,4.4] #<2-> implement session bind [DONE,4.4] #<2-> implement channel epoch numbers [DONE,4.4] #<2-> implement interface discovery [DONE(linux/conf),4.4] #<3-> _blue_implement test cases_ [WIP(isn't it always?... $\smiley$)] #<4-> _red_implement fd-passing in socket-wrapper_ [WIP (Anoop CS, obnox)] #<4-> _red_implement lease break replay_ [WIP (obnox, Günther, metze)] #<4-> _red_integrate Multi-Channel with clustering (CTDB)_ [WIP] %%% ==== Multi-Channel $\in$ Samba : Details from @smbXsrv.idl@ ==== %%% %%% <[block]{for @MSG\_SMBXSRV\_CONNECTION\_PASS@} %%% <[code] %%% typedef struct { %%% NTTIME initial_connect_time; %%% GUID client_guid; %%% hyper seq_low; %%% DATA_BLOB negotiate_request; %%% } smbXsrv_connection_pass0; %%% [code]> %%% [block]> %%% %%% ==== Multi-Channel $\in$ Samba : Details from @smbXsrv.idl@ ==== %%% %%% %%% <[block]{layering before} %%% <[code] %%% smbXsrv_session %%% ->smbXsrv_connection %%% [code]> %%% [block]> %%% %%% <[block]{layering now} %%% <[code] %%% smbXsrv_session %%% ->smbXsrv_client %%% ->smbXsrv_connections\[array\] %%% ->smbXsrv_channels\[array\] %%% -> smbXsrv_connection %%% [code]> %%% [block]> %%% ==== Multi-Channel $\in$ Samba : Status ==== %%% %%% <[block]{WIP code} %%% * @git://git.samba.org/obnox/samba/samba-obnox.git@ %%% ** branch: @master-multi-channel-obnox@ %%% * @git://git.samba.org/gd/samba/.git@ %%% ** branch master-multichannel %%% [block]> %%%==== Multi-Channel $\in$ Samba: TODOs ==== %%% %%%* Replay lease breaks upon channel failure (server $\rightarrow$ client) %%%* teach socket\_wrapper fd-passing ( ==> selftest...) %%%* clustering integration (CTDB) ==== selftest <== fd-passing $\in$ socket-wrapper ==== +<2->{ # untangle @socket\_info\_fd@ from @socket\_info@ # array (@sockets@) of @socket\_infos@ instead of linked list # protect structures from concurrent access by ptread mutexes #* @sockets@, @sockets[i]@, @socket\_fds@, @first\_free@ #* use process shared robust mutexes where indicated # put @sockets@ list and @first\_free@ index into a shared storage #* use a file, mmap into each user of swrap (like @tdb@) #* these are the parts to use robust mutexes for # implement fd-passing: #* create a pipe #* pass one end of pipe along with original fds array #* sender writes array of indexes to sockets array into pipe #* receiver reads indexes from pipe and creates new socket\_fd structures using fds and indexes } ==== Multi-Channel $\in$ Samba: Lease Break Replay ==== * oplock/lease break request: server --> client * protection with channel sequence numbers not available! * need to track sent but not acked break requests * declare channel dead if not acked for a while and resend (replay) them over other channels (if any) * if we don't do that ... multi-channel may eat your data \frownie ==== Multi-Channel $\in$ Samba: Lease Break Replay ==== * have send\_queue, add ack\_queue * use SIOCOUTQ ioctl on the tcp socket: \\ % unsent data in the socket send queue ==== Multi-Channel $\in$ Samba: Lease Break Replay ==== <[code] (left the host) +------ acked_byte +---sent_bytes | | v v +++++++++++++++++++++++++++++++++++++++++ |p1-----|p2------|p3-----|p4-----|p5----| ^ ^ | | | +-----p3.ack.last_byte | +--- p1.ack.last_byte [code]> ==== Multi-Channel $\in$ Samba: Lease Break Replay ==== <[block]{WIP code} * @git://git.samba.org/obnox/samba/samba-obnox.git@ ** branch: @master-multi-channel-obnox@ * @git://git.samba.org/gd/samba/.git@ ** branch @master-multichannel@ [block]> ==== Multi-Channel $\in$ Samba : Clustering/CTDB ==== +<2->{ <[block]{Special considerations} * channels of one session only to one node ! * do not bind connections to CTDB public IPs (can move)! * ==> !add static IPs on public interfaces! \\ % use these for interface discovery [block]> } [frame]>