Need ideas about ntoskrnl (WITH ideas)

Damjan Jovanovic dj015 at yahoo.com
Thu Sep 8 01:13:34 CDT 2005



--- Vitaliy Margolen <wine-devel at kievinfo.com> wrote:

> First let me thank everyone helping me and Ivan to
> get ntoskrnl moving
> forward.
> 
> Summary of all the ideas and requirements:
> 1. wineserver should not be used to run drivers
> (absence of required
>    functionality, single threaded, stability issues)
> 2. ntoskrnl should be used to run drivers only and
> link to ntdll and not the
>    other way around. Unless we implement int 0x2e
> handling and replace
>    wineserver with ntoskrnl. (this is wine not
> reactos, ntdll already has all
>    the required functionality)

Using int 0x2e and other Linux-specific hacks like
kernel modules, limits portability of Wine, you'll
never get it working on Windows itself.

> 3. Drivers can not be ran in a process that requires
> them (drivers we concerned
>    about keep information that is shared between
> different processes, drivers
>    require their own environment that is different
> from a user process
>    environment, ntdll has no means nor should it
> have them to run a driver)

I am worried about performance. When you change from a
real driver to a separate process with IPCs, your
response time goes from function calls to context
switches.

> And that's it.
> 
> This validates original design by Ivan Leo Puoti for
> ntoskrnl. Now, there are
> still unanswered questions:
> 1. How to talk to ntoskrnl (directly from ntdll or
> through wineserver).

You can't use plain UNIX IPC's or sockets, since when
you call ReadFile(), the driver doesn't know that the
other end of its socket received a read(). You need a
protocol that sends the driver a message then does a
read() on the socket. Either way ntdll needs to
changed.

For performance reasons, I would reduce the number of
processes each call has to go through, so avoid going
through wineserver.

> 2. How to get information required for ntoskrnl from
> wineserver.
> 3. How to identify this is a device operation and
> not a file/named pipe/mail
>    slot, etc.

I've wanted this functionality for a while now. Look
at ntdll/cdrom.c for a (really ugly) way that the
current CDROM driver uses (it does a linear search
through a table of HANDLE / fd structures). The only
clean solution I see is to store more data in the
wineserver regarding each handle.

> I know folks you have more ideas, keep them coming.

I'm attaching a few possible solutions I mailed to
Alexandre a few months ago (he ignored them :-( ).

> Vitaliy Margolen
> 
> 
> 

Bye
Damjan


	
		
______________________________________________________
Click here to donate to the Hurricane Katrina relief effort.
http://store.yahoo.com/redcross-donate3/
-------------- next part --------------
About STI

STI (Still Image Interface) is the still image system used on Windows 98 onwards. WIA, found on Windows XP, uses STI and provides more functionality.

So what does it do? Well, previously, image acquisition looked like so



               Application (eg. Imaging)
                      |
                      |
                      v
               TWAIN frontend (part of Windows)
                      |
                      |
                      v
               TWAIN backend (written by scanner manufacturer)
                      |
                      |
               (other software, kernel-mode drivers, etc.)
                      |
                      |
                      v
                   Scanner

Note how arrows go down only. So scanning was a pull protocol, with an application requesting an image from TWAIN, and requests going all the way down to the scanner.

So how do all the buttons you find on scanners nowdays work? STI allows scanning to be done via a push protocol:

        +----->Application (eg. Imaging)
        |             |
        |             |
        |             v
        |      TWAIN frontend (part of Windows)
        |             |
        |             |
        |             v
        |      TWAIN backend (written by vendor)
        |             |
        |             |
        |             v
        +------------STI (written by microsoft)
                      | ^
                      | |
                      v |
                    STI minidriver (written by vendor)
                      | ^
User mode             | |
-------------------   | |
Kernel mode           | |
                      v |
                    STI port driver (written by vendor or by Microsoft)
                      | ^
                      | |
                      v |
                    Scanner


STI does several things:
* An application can register itself with STI to handle scanner events (like button presses). This process is permanent (it's stored in the registry) until the application explicitly unregisters. The application can handle a particular scanner event, or all scanner events. If no application is registered, the event is ignored; if multiple applications are registered for the event, STI asks to user to pick one (and it doesn't store the picked application anywhere). The application is then started with command line parameters that indicate the device and the event.
* It loads the minidrivers when a known device is plugged in. A device becomes known through the still image class installer which registers the device with STI when its drivers are installed. The minidrivers are used by the TWAIN backend (or other software) to access the scanner, and they provide notification of scanner events. Minidrivers in turn communicate with kernel-mode drivers using CreateFile(), DeviceIoControl(), WriteFile() and ReadFile().
* A still-image event monitor (STIMON.EXE on Windows 98, STISVC.EXE on Windows 2000) needs to be running to load minidrivers when new scanners are plugged in and act when scanner events occur.
* Some kernel mode drivers are written by Microsoft, others have to be written by the vendor. On Windows 2000, USBSCAN.SYS, SCSISCAN.SYS and SERSCAN.SYS are there; Windows 98 doesn't have USBSCAN.SYS so vendors provide their own driver that's used by their STI minidriver. The interface to these drivers is well-documented :-) and can easily be forwarded through libusb on Linux (already been done, in tests).

STI is written in COM (yes it's ugly).
There are 4 classes used:

IStillImage - STI.DLL creates an object of this class with the StiCreateInstance() function. Apps use this to register and unregister themselves with STI, it can be used to view the plugged-in devices, get and set registry information for a particular device, and to create instances of IStiDevice.

IStiDevice - the interface through which you access a device, and register for device events.

IStiUSD - minidrivers implement this, it's mostly the same as IStiDevice (without any device event handler lists).

IStiDeviceControl - allows minidrivers to query the mode in which they were open and the name of their device port.

Plans for Wine
--------------
It would be tempting to implement all classes with generic implementations, but that's impossible. The drivers for my  scanner rely on the vendor-supplied minidriver to do something special, replacing it with a generic minidriver breaks everything.

Vendor minidrivers must therefore be loaded. So here comes the problem: when the minidriver tries to open USBSCAN.SYS, what then? As it is, the device path "\\.\USBSCAN.SYS" will parse as a VxD in wine, and thus fail. Since VxD's do not implement ReadFile() and WriteFile(), or the correct range of IOCTL codes, they cannot be used to implement wine's own USBSCAN.SYS. On ReactOS, on the other hand, it would be possible to do a kernel-mode USBSCAN.SYS, but on wine?

It would be possible to patch (more like hack apart, really) CreateFile() and/or NtCreateFile() to deal with USBSCAN.SYS specifically by allocating a special handle, and ReadFile() WriteFile() and DeviceIoControl() then checking for that handle, and invoking special behaviour that converts the data received into libusb requests. This works - the only reason it hasn't been done is that it requires substantial changes to the wineserver, and some people on the winedevel mailing list were threatening mutiny... While in my opinion the changes would be beneficial (DeviceIoControl() at the moment only works on CD-ROMS, ha ha), I'll leave that to some other brave soul.

My current line of thought is that things must be hacked far more locally to STI, ie. the minidrivers when loaded, must have their DLL imports patched to route CreateFile(), CloseHandle(), ReadFile(), WriteFile() and DeviceIoControl() through some internal functions that selectively fall back on either libusb or the real functions in KERNEL32.DLL. It is possible, just so ugly it's unbelievable, but it works...

After some thought, here's what I came up with:
* NtDeviceIoControl() or whatever it's called needs to be patched a bit. It should use DEVICE_TYPE_FROM_CTL_CODE() and demultiplex which function to call from that; so if it's a CDROM/DVDROM call CDROM_DeviceIoControl(), if it's a FILE_DEVICE_USB_SCAN call for example USBSCAN_DeviceIoControl().
* As for creating the handle, patch CreateFile() and/or NtCreateFile() to deal with USBSCAN.SYS in particular (proper handling requires more work, but it's a start). They should use the pipe() call to create a pipe, fork() or in some other way create a process / thread, 

NO! Cannot read() on a pipe() - how would the other end know when it has to usb_bulk_read(), and how many bytes to send?

Must implement this properly...

---

I've been working on an STI (Still Image system) implementation for wine, and I either need good ideas, or changes to some wine fundamentals (NTDLL and wineserver). I'd like your opinion.

The part of STI that won't work in wine's current state, is the STI minidriver, which is a DLL, provided by the hardware vendor, that allows applications to access the still image device:

             Minidriver (written by hardware vendor)
                  |
User mode         |
-----------       |
Kernel mode       |
                  |
                  v
         Kernel-mode driver (written by hardware vendor / Microsoft)
                  |
                  v
               Device

(I'm using USB as an example; STI works with SCSI, serial and infrared hardware too.)

The minidriver communicates with the kernel-mode driver (which is named, for example, "USBSCAN.SYS") using CreateFile() then DeviceIoControl(), ReadFile() and WriteFile(). The problems are:

[1] CreateFile() thinks "\\.\USBSCAN" is a VxD, and so immediately fails. Adding a VxD of this name won't work, since ReadFile() and WriteFile() are not supported by VxD's anyway.
[2] DeviceIoControl(), ReadFile() and WriteFile() need to be implemented either using libusb, or Linux-specific ioctl()'s that work on the /proc filesystem (of course, hardware on other busses is a different story). At present, NtDeviceIoControlFile() calls CDROM_DeviceIoControl(), and NtReadFile() and NtWriteFile() call read() and write(), which isn't very useful.

Possible solutions, in increasing order of preference:

1. GENERIC MINIDRIVER
---------------------
Make a minidriver that implements the IStiDevice interface and uses libusb directly, skipping USBSCAN.SYS.

Disadvantages:
* Tried it, doesn't work with my scanner and about 30 others (the vendor's minidriver does something special)

2. KERNEL MODULE
----------------
Implement a kernel module that would accept read() write() and ioctl() calls, and deal with them like USBSCAN.SYS would. NtDeviceIoControlFile() gets modified to call ioctl() when IOCTL_CODE_TO_DEVICE_TYPE(dwControlCode) == DEVICE_TYPE_USB_SCAN.

Advantages:
* Only CreateFile() and NtDeviceIoControlFile() need to be changed

Disadvantages:
* completely Linux-specific (zero portability)
* ioctl codes inconsistent with Linux kernel numbering (unlikely to be included in kernel tree)
* cannot be written by me (no experience coding kernel modules)
* problems porting to different Linux kernel versions (eg. the SANE project had a scanner kernel module; it died out with 2.4 kernels).
* needs several kernel modules for USB, SCSI, serial and IR.

3. DLL PATCHING
---------------
Patch DLL imports for the minidriver so CreateFile(), ReadFile(), WriteFile() and DeviceIoControl() get dynamically linked to alternative implementations that use libusb functions.

Advantages:
* no changes to existing wine code

Disadvantages:
* wine doesn't support DLL patching
* if the minidriver starts doing asynchronous I/O ...
* many versions of each function, for USB, SCSI, etc.
* it's a hack - doesn't solve the real problem

4. TYPED HANDLES
----------------
Modify HANDLE to somehow store handle type, or else function pointers to functions used on handles, like reading, writing and ioctl's. NTDLL functions like NtWriteFile(), NtReadFile() and NtDeviceIoControlFile() should use the handle type to demultiplex the I/O request to the correct function. So for example when NtDeviceIoControlFile() is called on a handle of type (say) WINE_HANDLE_USB_SCAN, the function UsbScanDeviceIoControl() is called, when it's called on a handle of type WINE_HANDLE_CDROM then CDROM_DeviceIoControl() is called. It would help too if some generic data (eg. void*) could be associated with each handle, so code managing a handle family could store some internal data there.
Personally I advise:
	struct HandleData {
		VOID (*Close)(HANDLE handle);
		DWORD (*Ioctl)(HANDLE handle, /* Other DeviceIoControl() parameters */);
		DWORD (*Read)(HANDLE handle, LPVOID *buffer, DWORD size);
		DWORD (*Write)(HANDLE handle, LPVOID *buffer, DWORD size);
		VOID *internalData;
	};
	#define HANDLE_TO_HandleData(h) ...

Advantages:
* Flexible; easy to add other hardware support in general
* For STI: SCSI, serial and IR easier to add
* Less hacks in other wine code (have you seen "struct cdrom_hash" and the comment "This should be removed when a proper device interface is implemented" in dlls/ntdll/cdrom.c?)

Disadvantages:
* the winedev mailing list wasn't very thrilled last time I mentioned it ("device drivers don't belong in wine")
* might require changes to the wineserver (does it?)


I personally like option 4 (typed handles), but since you maintain wineserver, and know wine better than anyone, what do you think? Any other ideas?


More information about the wine-devel mailing list