Summary of MSRPC / DCE/RPC interoperability / Applicability of FreeDCE for Wine (DCOM)

[Luke Kenneth Casson Leighton]

hi, in case you looked at the number of emails exchanged on FreeDCE
and freaked out, here's a summary.  if you feel that there are any
relevant points missing or incorrect, please _do_ speak up.


	[p.s. this message not sent to samba mailing lists due to 
	censorship decision by samba team in effect since
	15th dec 2004.	if anyone believes this message to be
	relevant to samba development please kindly forward
	it to the samba lists, checking first to see if you
	are the first person to do that.  or simply feel free
	to add cc on any comments to any samba lists you feel
	are appropriate.  please _do_ remember to remove my email
	address from any cc's to samba lists because it is likely
	to trigger the samba team censorship filters.
	thank you]


1) initial query raised:

	"freedce suitable for wine, esp. dcom!"

2) comments made (corrections from mark, rob and wez):

	"freedce doesn't have anything _like_ the capability
	 required for DCOM - freedce's DCOM code is 'stubs'"

3) confusion resulted, clarification process began (rob, luke).
   conclusion:

a) "freedce is almost identical to MSRPC, and is the back-end
    to rpcrt4.dll"

b) "MSRPC as implemented in rpcrt4.dll has NOTHING to do with DCOM:
    DCOM compiles to code that ***UTILISES*** rpcrt4.dll"

   (this requires further explanation, see below).

c) Porting of / Plugging in FreeDCE to rpcrt4.dll could be done
   "a la Apache Runtime (APR)", i.e. wrapper functions - maybe even
   just macros in most cases.

d) extending FreeDCE to support rpcrt4.dll is easy and almost
   boring - even RpcImpersonateClient would be trivial to add

   (trivial from a Wine perspective - but see below for implications)

e) lkcl suggested porting FreeDCE to use Wine's threads because
   dcethreads (posix draft 4 threads) have to diiiieeee.

   this can be done independently of Wine and there is an incentive
   _outside_ of Wine to _get_ it done because the dependency on
   posix draft 4 threads is so "last decade".

f) modifying and utilising FreeDCE to understand Win32 "type string
   format" tables is, with the help of muddle (a program written
   by matthew chapman that understands exactly those tables in
   Win32 DLLs and EXEs) going to be a pretty trivial task.

   (see below for more details here)

g) extension of DCE/RPC syntax to support MSRPC syntax
   is infinitely more straightforward than rewriting an IDL
   compiler from scratch.
   
   (see below)

... still with me so far? :)  now here's the catch, and the catch
to the catch :)

the main counterargument raised to utilising freedce was not
so much a technical argument, but was a resources-related and
maintenance argument.

	'why bother, when the approach presently being taken
	will "work"'?


to answer that, i have to drop out of "summary" mode and into
a mode of adding to and also reiterating [some of the] points
raised last week.


1) it'll work - _eventually_ - and not fully [codeweavers present
   aims are for binary and type string interoperability only,
   and for only a subset of MSRPC to keep the time, development
   and maintenance costs down]

2) if you ever intend to aim for full interoperability with MSRPC,
   i recommend that you investigate how "full" pointers are
   implemented, because i know of a [proprietary] company that
   implemented that, after first only doing unique pointers,
   and adding "full" pointers QUADRUPLED the amount of code that
   they ended up with.

3) why aim to implement (from scratch) only 25% of the full solution 
   (i.e. leaving out network interoperability by implementing
   your own non-NDR wire format, and not even implementing the
   full capabilities of the DCE/RPC IDL language) when a 95 to 98%
   feature-complete and free-software-license-compatible solution
   already exists?

   madness!  madness, i say!

4) by utilising FreeDCE in Wine, you can kill two birds (actually three
   or maybe four) with one stone - two of which were aimed for
   with the initial DCOM work (DCOM binary interoperability at
   the Win32 and "type string" table levels) and
   two more are gained "by accident" - network interoperability
   and the ability to run MSRPC-based applications.

5) the effort expended so far is most certainly not "wasted" by
   adopting FreeDCE.  if the time and effort expended so far had
   not been spent, how would it then be possible to appreciate
   the scope and scale of adapting and maintaining FreeDCE
   integrated to suit Wine?

   remember, i went through the same thing - i implemented
   from scratch HAND-CRAFTED marshalling and unmarshalling
   code in NT Domain interoperability code - almost 100,000
   lines over three years - before deciding it was insane and
   seeking alternatives [and i found FreeDCE].

6) if you understand the present Wine rpcrt4.dll code, FreeDCE is
   _so_ similar that the issue of "it's so different that
   codeweavers and the wine developers couldn't _possibly_ adapt
   to it" is just absolute nonsense and a complete non-issue:
   it _is_ the same code (OSF DCE 1.1 "reference" implementation)
   
   it's just that in rpcrt4.dll, DCE 1.1 has been "microsofted",
   and in FreeDCE, DCE 1.1 has been "gnu'd" and "autoconf'd".


the question therefore, in my mind, turns on its head:

	"how _can_ you turn such an opportunity to save time
	 (and money, and effort) down?"


summary (well, sort of summary) ends here.

the rest of this message is notes about the two. three points above.
[noooobody expects the spanish inquisitionnn.  the key element
 is surpriiise.  surprise and ... the _twooo_ key elements .... ]


DCOM is nothing to do with MSRPC

regarding 3b) - this is where the initial confusion arose in 1).
DCOM is, as rob kindly clarified, an object-orientated "thing"
that uses MSRPC for its transport.  it is a programming concept,
not a networking API.

in microsoft's c++ implementation of DCOM, once the c++ compiler has
done its job, no evidence of the fact that it was DCOM remains:
everything is compiled down onto MSRPC interfaces [including certain
services that are required as part of DCOM].

therefore, it is both easy to confuse rpcrt4 as an implementation
of "DCOM" (which it is not, it is an implementation of MSRPC) and
also it is easy to forget that DCOM expects some interfaces
(including the c++ APIs if you want your c++ applications to
utilise DCOM) and services (e.g. the additional service in
the endpoint mapper) to exist.

and, according to documentation written by microsoft and available
online at the open group's site, one of those services is a Win32
registry (yuk).

oops :)

for which there exists samba tng code that implements the MSRPC
side of a Win32 registry, btw, which could easily be combined with
Wine's registry to complete a full proper job.


RpcImpersonateClient and ImpersonateNamedPipeClient

these functions can be implemented such that FreeDCE will transfer
security contexts about (to and from Win32 applications that ask for
them).  the process of adding the two above functions to Wine is
very straightforward: pass a few blobs around, such that they can
be passed to SeAccessCheck amongst other things.

the _implications_ of adding them are that ReactOS must actually
understand them, make use of them, etc. [otherwise, you needn't
bother - just implement stubs]

and _that_ means that the integration with and adoption of
Samba or Samba TNG's architecture and NT Services needs to
be accelerated.  the Samba Team will need to be brought up to
speed on how this is achieved.

you may run into difficulties, there, however.  their general
ethos is to keep complete control over all projects that they
are involved with or need to make use of (emphasis on _they_
need), and where this is not possible, they tend to reimplement
from scratch any components that they require (kerberos servers,
ldap servers, that sort of thing).

recent developments, however, tend to suggest that this ethos
is changing - but the proof of the pudding is always in the eating:
their track record on cooperation with other projects is not good.

you may therefore find it easier to work with the samba tng
architecture for the initial implementation of these two
functions, because samba tng is already split into easily
manageable and understandable portions behind APIs that match
or mirror the samba tng team's "best-guess" of the underlying
(or unpublished) Win32 apis.

and, as frequently happens, once a concept has been demonstrated
as viable in samba tng, it is often adopted by the samba team -
so you need not have any concerns about utilising the samba tng
architecture.  consider it a "proving ground".


NDR "type string" interoperability - f)

compared to writing an interpreter from scratch that understands
these tables, adapting FreeDCE to understand MSRPC tables
in DLLs and EXEs is most certainly trivial.

NDR "type strings" - aka in DCE/RPC formats as "AST", is effectively
a byte-code interpreted language, one in which the size of data
structures and the type and offsets of members of structs is embedded,
amongst almost a hundred or so other things.

remember, binary-interoperability with _linux_ is totally irrelevant:
you can just recompile the source code, link against static libraries,
blah blah, sorted.

so adapting FreeDCE for win32 "type string" interoperability
it's a matter of:

1) changing some constants and alignments where necessary,
   to match those in MSRPC (analysing muddle and freedce)

2) improving muddle to correctly identify the data structures
   and their relative positions in the DLLs / EXEs - at present,
   muddle "guesses" some of the table's locations (eek!)

and as an aside / prerequisite:

3) correctly duplicating any MSRPC extensions, including adding
   in support for "complex" expressioning.


MSRPC interoperability - g)

   "interface" inheritance has already been added (by wez).

   "expressions" needed for size_is(complexthing) have already
   been added [needs more work and needs "outsourcing" to a
   function]

   range(n-n) needs to be added (trivial to do).

   

-- 
--
<a href="http://lkcl.net">http://lkcl.net</a>
--