27 Jan
2022
27 Jan
'22
9:50 p.m.
Over the years, Wine prefixes have gotten bigger and bigger, for a
number of reasons. Creating a new Wine prefix for each application is
still the current recommendation, as despite the best efforts of Wine
developers, some applications still require system-wide workarounds.
This leads to significant bloat for each application installed. With a
MinGW build of Wine without Mono or Gecko, new 32-bit prefixes are over
150 MB, and new 64-bit prefixes are over 300 MB. The vast majority of
these files are byte-for-byte identical to Wine's central DLL copies.
This patch set implements reflink support in Wine via the
copy_file_range syscall. The reasons for selecting copy_file_range over
FICLONE are outlined in patch 2. A previous unpublished version of this
patch set used FICLONERANGE, but it was less convenient to use from
setupapi and has inferior system support.
When reflink is supported by the underlying filesystem, new Wine prefix
sizes with Mono and Gecko disabled are reduced to less than 1 MB. The
resulting Wine prefix is byte-for-byte identical to one created without
reflink, but occupies less space on disk. If hard links or symlinks were
used, if an application such as winetricks writes to a system file, it
would overwrite the central copy. With reflink, the file blocks will be
transparently copied by the Linux kernel so that each Wine prefix can be
independent.
Some files cannot be deduplicated in the current Wine system, as they
are dynamically generated during the Wine prefix installation process.
These include 16-bit fake DLLs and manifest files. In theory, it should
be possible to pre-generate these files, but considering the Wine prefix
size is already reduced to less than 1 MB, the extra space savings are
likely not worth the effort.
v1->v2: Fix/improve errno handling.
v2->v3: Rebase.
Alex Xu (Hello71) (5):
ntdll: add support for IOCTL_COPYCHUNK.
kernelbase: use IOCTL_COPYCHUNK in CopyFile*
setupapi: Use IOCTL_COPYCHUNK, avoid buffering whole file
lmshare.h: define STYPE_TEMPORARY
kernel32/tests: add IOCTL_COPYCHUNK test
configure | 6 ++
configure.ac | 1 +
dlls/kernel32/tests/file.c | 137 +++++++++++++++++++++++++
dlls/kernelbase/file.c | 44 ++++-----
dlls/ntdll/unix/file.c | 121 +++++++++++++++++++++++
dlls/setupapi/fakedll.c | 198 ++++++++++++++++++++-----------------
include/config.h.in | 3 +
include/lmshare.h | 11 ++-
include/winioctl.h | 34 +++++++
9 files changed, 434 insertions(+), 121 deletions(-)
--
2.35.0
27 Jan
27 Jan
9:50 p.m.
New subject: [PATCH v3 1/5] ntdll: add support for IOCTL_COPYCHUNK.
This API really sucks; it literally just passes through the raw SMB
request. However, there are some benefits over
FSCTL_DUPLICATE_EXTENTS_TO_FILE:
1. it's easier to use from setupapi than FSCTL_DUPLICATE_EXTENTS_TO_FILE
which can't plausibly be emulated.
2. on Windows, IOCTL_COPYCHUNK is already called (indirectly) from
CopyFile on Windows. see e.g.
https://www.ghisler.ch/board/viewtopic.php?t=43945.
3. copy_file_range allows kernel acceleration if reflink isn't possible.
4. copy_file_range is supported on FreeBSD.
---
configure | 6 ++
configure.ac | 1 +
dlls/ntdll/unix/file.c | 121 +++++++++++++++++++++++++++++++++++++++++
include/config.h.in | 3 +
include/winioctl.h | 34 ++++++++++++
5 files changed, 165 insertions(+)
diff --git a/configure b/configure
index 7760e43b8e9..78e502c28d6 100755
--- a/configure
+++ b/configure
@@ -19717,6 +19717,12 @@ esac
ac_save_CFLAGS="$CFLAGS"
CFLAGS="$CFLAGS $BUILTINFLAG"
+ac_fn_c_check_func "$LINENO" "copy_file_range"
"ac_cv_func_copy_file_range"
+if test "x$ac_cv_func_copy_file_range" = xyes
+then :
+ printf "%s\n" "#define HAVE_COPY_FILE_RANGE 1" >>confdefs.h
+
+fi
ac_fn_c_check_func "$LINENO" "epoll_create"
"ac_cv_func_epoll_create"
if test "x$ac_cv_func_epoll_create" = xyes
then :
diff --git a/configure.ac b/configure.ac
index 181047ef0d8..de2f810b4ac 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1957,6 +1957,7 @@ dnl **** Check for functions ****
ac_save_CFLAGS="$CFLAGS"
CFLAGS="$CFLAGS $BUILTINFLAG"
AC_CHECK_FUNCS(\
+ copy_file_range \
epoll_create \
fstatfs \
futimens \
diff --git a/dlls/ntdll/unix/file.c b/dlls/ntdll/unix/file.c
index a29b5cbb980..cedb57e8f2f 100644
--- a/dlls/ntdll/unix/file.c
+++ b/dlls/ntdll/unix/file.c
@@ -347,6 +347,7 @@ NTSTATUS errno_to_status( int err )
#ifdef ETIME /* Missing on FreeBSD */
case ETIME: return STATUS_IO_TIMEOUT;
#endif
+ case ENOMEM: return STATUS_NO_MEMORY;
case ENOEXEC: /* ?? */
case EEXIST: /* ?? */
default:
@@ -5694,6 +5695,122 @@ NTSTATUS WINAPI NtWriteFileGather( HANDLE file, HANDLE event,
PIO_APC_ROUTINE ap
}
+static NTSTATUS netfs_DeviceIoControl( HANDLE handle, HANDLE event, PIO_APC_ROUTINE apc,
void *apc_context,
+ IO_STATUS_BLOCK *io, ULONG code, void *in_buffer,
ULONG in_size,
+ void *out_buffer, ULONG out_size )
+{
+ /* wine extension: support IOCTL_COPYCHUNK even on local files (no way to tell with
+ * copy_file_range anyways) */
+ NTSTATUS status;
+
+ switch (code)
+ {
+ case IOCTL_PREPARE_COPYCHUNK:
+ {
+ /* wine extension: out_buffer only needs to be big enough to hold a HANDLE */
+ if (out_size < sizeof(HANDLE))
+ {
+ io->Information = 0;
+ status = STATUS_BUFFER_TOO_SMALL;
+ break;
+ }
+ io->Information = sizeof(HANDLE);
+ *(HANDLE *)out_buffer = handle;
+ status = STATUS_SUCCESS;
+ break;
+ }
+
+ /* wine extension: support IOCTL_COPYCHUNK with chunk sizes greater than 1 MB */
+ case IOCTL_COPYCHUNK:
+ {
+ static const SIZE_T buffer_size = 65536;
+ SRV_COPYCHUNK_COPY *cc = in_buffer;
+ SRV_COPYCHUNK_RESPONSE *ccr = out_buffer;
+ int src_fd, dst_fd, src_needs_close, dst_needs_close;
+ void *buffer = NULL;
+ BOOL fallback = FALSE;
+ if (in_size < sizeof(*cc))
+ {
+ status = STATUS_INVALID_PARAMETER;
+ break;
+ }
+ if (out_size < sizeof(*ccr))
+ {
+ status = STATUS_BUFFER_TOO_SMALL;
+ break;
+ }
+ status = server_get_unix_fd( handle, FILE_WRITE_DATA, &dst_fd,
&dst_needs_close, NULL, NULL );
+ if (status) break;
+ status = server_get_unix_fd( (HANDLE)(ULONG_PTR)cc->SourceFile.ResumeKey,
FILE_READ_DATA, &src_fd, &src_needs_close, NULL, NULL );
+ if (status)
+ {
+ if (dst_needs_close) close( dst_fd );
+ break;
+ }
+ io->Information = sizeof(*ccr);
+ ccr->TotalBytesWritten = 0;
+ for (ccr->ChunksWritten = 0; ccr->ChunksWritten < cc->ChunkCount;
ccr->ChunksWritten++)
+ {
+ off_t off_in = cc->Chunk[ccr->ChunksWritten].SourceOffset.QuadPart;
+ off_t off_out =
cc->Chunk[ccr->ChunksWritten].DestinationOffset.QuadPart;
+ size_t len = cc->Chunk[ccr->ChunksWritten].Length;
+#ifdef HAVE_COPY_FILE_RANGE
+ if (!fallback)
+ {
+ ssize_t res = copy_file_range( src_fd, &off_in, dst_fd, &off_out,
len, 0 );
+ if (res == -1)
+ {
+ if (errno == EXDEV || errno == EINVAL || errno == ENOSYS) fallback =
TRUE;
+ else goto copychunk_out;
+ }
+ else
+ {
+ ccr->ChunkBytesWritten = ccr->TotalBytesWritten = res;
+ }
+ }
+ if (fallback)
+#endif
+ {
+ if (!buffer) buffer = malloc( buffer_size );
+ if (!buffer) goto copychunk_out;
+ ccr->ChunkBytesWritten = 0;
+ char *p = buffer;
+ ssize_t count = pread( src_fd, buffer, min( buffer_size, len ), off_in
);
+ if (count == -1) goto copychunk_out;
+ off_in += count;
+ while (count)
+ {
+ ssize_t res = pwrite( dst_fd, p, count, off_out );
+ if (res == -1) goto copychunk_out;
+ p += res;
+ count -= res;
+ off_out += res;
+ ccr->ChunkBytesWritten += res;
+ ccr->TotalBytesWritten += res;
+ }
+ }
+ if (ccr->ChunkBytesWritten != cc->Chunk[ccr->ChunksWritten].Length)
break;
+ ccr->ChunkBytesWritten = 0;
+ }
+ errno = 0;
+
+copychunk_out:
+ if (errno) status = errno_to_status( errno );
+ else status = STATUS_SUCCESS;
+ if (buffer) free( buffer );
+ if (src_needs_close) close( src_fd );
+ if (dst_needs_close) close( dst_fd );
+ break;
+ }
+ default:
+ status = STATUS_NOT_SUPPORTED;
+ }
+
+ io->u.Status = status;
+ return status;
+}
+
+
/******************************************************************************
* NtDeviceIoControlFile (NTDLL.@)
*/
@@ -5709,6 +5826,10 @@ NTSTATUS WINAPI NtDeviceIoControlFile( HANDLE handle, HANDLE event,
PIO_APC_ROUT
switch (device)
{
+ case FILE_DEVICE_NETWORK_FILE_SYSTEM:
+ status = netfs_DeviceIoControl( handle, event, apc, apc_context, io, code,
+ in_buffer, in_size, out_buffer, out_size );
+ break;
case FILE_DEVICE_BEEP:
case FILE_DEVICE_NETWORK:
status = sock_ioctl( handle, event, apc, apc_context, io, code, in_buffer,
in_size, out_buffer, out_size );
diff --git a/include/config.h.in b/include/config.h.in
index 9717dc17236..9150285b39e 100644
--- a/include/config.h.in
+++ b/include/config.h.in
@@ -37,6 +37,9 @@
/* Define to 1 if you have the <CL/cl.h> header file. */
#undef HAVE_CL_CL_H
+/* Define to 1 if you have the `copy_file_range' function. */
+#undef HAVE_COPY_FILE_RANGE
+
/* Define to 1 if you have the <CoreAudio/CoreAudio.h> header file. */
#undef HAVE_COREAUDIO_COREAUDIO_H
diff --git a/include/winioctl.h b/include/winioctl.h
index ae04c37a462..fbc0b80e359 100644
--- a/include/winioctl.h
+++ b/include/winioctl.h
@@ -331,6 +331,9 @@
#define FSCTL_PIPE_INTERNAL_TRANSCEIVE CTL_CODE(FILE_DEVICE_NAMED_PIPE, 2047,
METHOD_NEITHER, FILE_READ_DATA | FILE_WRITE_DATA)
#define FSCTL_PIPE_INTERNAL_READ_OVFLOW CTL_CODE(FILE_DEVICE_NAMED_PIPE, 2048,
METHOD_BUFFERED, FILE_READ_DATA)
+#define IOCTL_PREPARE_COPYCHUNK CTL_CODE(FILE_DEVICE_NETWORK_FILE_SYSTEM, 261,
METHOD_BUFFERED, FILE_ANY_ACCESS)
+#define IOCTL_COPYCHUNK CTL_CODE(FILE_DEVICE_NETWORK_FILE_SYSTEM, 262,
METHOD_BUFFERED, FILE_READ_ACCESS)
+
#define IOCTL_STORAGE_BASE FILE_DEVICE_MASS_STORAGE
#define IOCTL_STORAGE_CHECK_VERIFY CTL_CODE(IOCTL_STORAGE_BASE, 0x0200,
METHOD_BUFFERED, FILE_READ_ACCESS)
#define IOCTL_STORAGE_CHECK_VERIFY2 CTL_CODE(IOCTL_STORAGE_BASE, 0x0200,
METHOD_BUFFERED, FILE_ANY_ACCESS)
@@ -591,6 +594,37 @@ typedef struct RETRIEVAL_POINTERS_BUFFER {
} Extents[1];
} RETRIEVAL_POINTERS_BUFFER, *PRETRIEVAL_POINTERS_BUFFER;
+typedef struct _SRV_RESUME_KEY {
+ UINT64 ResumeKey;
+ UINT64 Timestamp;
+ UINT64 Pid;
+} SRV_RESUME_KEY, *PSRV_RESUME_KEY;
+
+typedef struct _SRV_REQUEST_RESUME_KEY {
+ SRV_RESUME_KEY Key;
+ ULONG ContextLength;
+ BYTE Context[1];
+} SRV_REQUEST_RESUME_KEY, *PSRV_REQUEST_RESUME_KEY;
+
+typedef struct _SRV_COPYCHUNK {
+ LARGE_INTEGER SourceOffset;
+ LARGE_INTEGER DestinationOffset;
+ ULONG Length;
+} SRV_COPYCHUNK, *PSRV_COPYCHUNK;
+
+typedef struct _SRV_COPYCHUNK_COPY {
+ SRV_RESUME_KEY SourceFile;
+ ULONG ChunkCount;
+ ULONG Reserved;
+ SRV_COPYCHUNK Chunk[1]; // Array
+} SRV_COPYCHUNK_COPY, *PSRV_COPYCHUNK_COPY;
+
+typedef struct _SRV_COPYCHUNK_RESPONSE {
+ ULONG ChunksWritten;
+ ULONG ChunkBytesWritten;
+ ULONG TotalBytesWritten;
+} SRV_COPYCHUNK_RESPONSE, *PSRV_COPYCHUNK_RESPONSE;
+
/* End: _WIN32_WINNT >= 0x0400 */
/*
--
2.35.0
9:50 p.m.
New subject: [PATCH v3 2/5] kernelbase: use IOCTL_COPYCHUNK in CopyFile*
---
dlls/kernelbase/file.c | 44 +++++++++++++++++-------------------------
1 file changed, 18 insertions(+), 26 deletions(-)
diff --git a/dlls/kernelbase/file.c b/dlls/kernelbase/file.c
index 576e03eb62b..f462b0caccd 100644
--- a/dlls/kernelbase/file.c
+++ b/dlls/kernelbase/file.c
@@ -497,24 +497,22 @@ BOOL WINAPI DECLSPEC_HOTPATCH AreFileApisANSI(void)
BOOL WINAPI CopyFileExW( const WCHAR *source, const WCHAR *dest, LPPROGRESS_ROUTINE
progress,
void *param, BOOL *cancel_ptr, DWORD flags )
{
- static const int buffer_size = 65536;
HANDLE h1, h2;
FILE_BASIC_INFORMATION info;
IO_STATUS_BLOCK io;
DWORD count;
BOOL ret = FALSE;
- char *buffer;
+ SRV_COPYCHUNK_COPY copychunk = {
+ .ChunkCount = 1,
+ .Chunk = { [0] = { .Length = 1UL<<30 } }
+ };
+ SRV_COPYCHUNK_RESPONSE copychunk_resp;
if (!source || !dest)
{
SetLastError( ERROR_INVALID_PARAMETER );
return FALSE;
}
- if (!(buffer = HeapAlloc( GetProcessHeap(), 0, buffer_size )))
- {
- SetLastError( ERROR_NOT_ENOUGH_MEMORY );
- return FALSE;
- }
TRACE("%s -> %s, %x\n", debugstr_w(source), debugstr_w(dest), flags);
@@ -522,14 +520,12 @@ BOOL WINAPI CopyFileExW( const WCHAR *source, const WCHAR *dest,
LPPROGRESS_ROUT
NULL, OPEN_EXISTING, 0, 0 )) == INVALID_HANDLE_VALUE)
{
WARN("Unable to open source %s\n", debugstr_w(source));
- HeapFree( GetProcessHeap(), 0, buffer );
return FALSE;
}
if (!set_ntstatus( NtQueryInformationFile( h1, &io, &info, sizeof(info),
FileBasicInformation )))
{
WARN("GetFileInformationByHandle returned error for %s\n",
debugstr_w(source));
- HeapFree( GetProcessHeap(), 0, buffer );
CloseHandle( h1 );
return FALSE;
}
@@ -545,7 +541,6 @@ BOOL WINAPI CopyFileExW( const WCHAR *source, const WCHAR *dest,
LPPROGRESS_ROUT
}
if (same_file)
{
- HeapFree( GetProcessHeap(), 0, buffer );
CloseHandle( h1 );
SetLastError( ERROR_SHARING_VIOLATION );
return FALSE;
@@ -557,28 +552,25 @@ BOOL WINAPI CopyFileExW( const WCHAR *source, const WCHAR *dest,
LPPROGRESS_ROUT
info.FileAttributes, h1 )) == INVALID_HANDLE_VALUE)
{
WARN("Unable to open dest %s\n", debugstr_w(dest));
- HeapFree( GetProcessHeap(), 0, buffer );
CloseHandle( h1 );
return FALSE;
}
- while (ReadFile( h1, buffer, buffer_size, &count, NULL ) && count)
+ /* wine extension: IOCTL_COPYCHUNK works even on local files */
+ ret = DeviceIoControl( h1, IOCTL_PREPARE_COPYCHUNK, NULL, 0,
+ ©chunk.SourceFile, sizeof(copychunk.SourceFile),
&count, NULL );
+ if (ret)
{
- char *p = buffer;
- while (count != 0)
- {
- DWORD res;
- if (!WriteFile( h2, p, count, &res, NULL ) || !res) goto done;
- p += res;
- count -= res;
- }
+ do {
+ ret = DeviceIoControl( h2, IOCTL_COPYCHUNK, ©chunk,
sizeof(copychunk),
+ ©chunk_resp, sizeof(copychunk_resp),
&count, NULL );
+ copychunk.Chunk[0].SourceOffset.QuadPart +=
copychunk_resp.TotalBytesWritten;
+ copychunk.Chunk[0].DestinationOffset.QuadPart +=
copychunk_resp.TotalBytesWritten;
+ } while (ret && copychunk_resp.TotalBytesWritten);
+ /* Maintain the timestamp of source file to destination file */
+ info.FileAttributes = 0;
+ NtSetInformationFile( h2, &io, &info, sizeof(info), FileBasicInformation
);
}
- ret = TRUE;
-done:
- /* Maintain the timestamp of source file to destination file */
- info.FileAttributes = 0;
- NtSetInformationFile( h2, &io, &info, sizeof(info), FileBasicInformation );
- HeapFree( GetProcessHeap(), 0, buffer );
CloseHandle( h1 );
CloseHandle( h2 );
return ret;
--
2.35.0
9:50 p.m.
New subject: [PATCH v3 3/5] setupapi: Use IOCTL_COPYCHUNK, avoid buffering whole file
---
dlls/setupapi/fakedll.c | 198 ++++++++++++++++++++++------------------
1 file changed, 108 insertions(+), 90 deletions(-)
diff --git a/dlls/setupapi/fakedll.c b/dlls/setupapi/fakedll.c
index a4b8df4dfb4..85c6f2500d8 100644
--- a/dlls/setupapi/fakedll.c
+++ b/dlls/setupapi/fakedll.c
@@ -19,8 +19,6 @@
*/
#include <stdarg.h>
-#include <fcntl.h>
-#include <sys/stat.h>
#include <unistd.h>
#define COBJMACROS
@@ -32,6 +30,7 @@
#include "winbase.h"
#include "winuser.h"
#include "winnt.h"
+#include "winioctl.h"
#include "winternl.h"
#include "wine/debug.h"
#include "wine/list.h"
@@ -64,8 +63,6 @@ static const unsigned int file_alignment = 512;
static const unsigned int section_alignment = 4096;
static const unsigned int max_dll_name_len = 64;
-static void *file_buffer;
-static SIZE_T file_buffer_size;
static unsigned int handled_count;
static unsigned int handled_total;
static WCHAR **handled_dlls;
@@ -169,10 +166,11 @@ static int is_valid_ptr( const void *data, SIZE_T size, const void
*ptr, SIZE_T
}
/* extract the 16-bit NE dll from a PE builtin */
-static void extract_16bit_image( IMAGE_NT_HEADERS *nt, void **data, SIZE_T *size )
+static void extract_16bit_image( void **data, SIZE_T *size )
{
DWORD exp_size, *size_ptr;
- IMAGE_DOS_HEADER *dos;
+ IMAGE_DOS_HEADER *dos = *data;
+ IMAGE_NT_HEADERS *nt = (IMAGE_NT_HEADERS *)((char *)*data + dos->e_lfanew);
IMAGE_EXPORT_DIRECTORY *exports;
IMAGE_SECTION_HEADER *section = NULL;
WORD *ordinals;
@@ -205,60 +203,81 @@ static void extract_16bit_image( IMAGE_NT_HEADERS *nt, void **data,
SIZE_T *size
}
}
-/* read in the contents of a file into the global file buffer */
+/* open and check a fake dll file */
/* return 1 on success, 0 on nonexistent file, -1 on other error */
-static int read_file( const WCHAR *name, void **data, SIZE_T *size )
+static int read_file( const WCHAR *name, HANDLE *h )
{
- struct stat st;
- int fd, ret = -1;
- size_t header_size;
+ char file_buffer[4096];
+ DWORD bytes_read;
IMAGE_DOS_HEADER *dos;
IMAGE_NT_HEADERS *nt;
- const size_t min_size = sizeof(*dos) + 32 +
- FIELD_OFFSET( IMAGE_NT_HEADERS, OptionalHeader.MajorLinkerVersion );
- if ((fd = _wopen( name, O_RDONLY | O_BINARY )) == -1) return 0;
- if (fstat( fd, &st ) == -1) goto done;
- *size = st.st_size;
- if (!file_buffer || st.st_size > file_buffer_size)
- {
- VirtualFree( file_buffer, 0, MEM_RELEASE );
- file_buffer = NULL;
- file_buffer_size = st.st_size;
- if (NtAllocateVirtualMemory( GetCurrentProcess(), &file_buffer, 0,
&file_buffer_size,
- MEM_COMMIT, PAGE_READWRITE )) goto done;
- }
+ *h = CreateFileW( name, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL
);
+ if (*h == INVALID_HANDLE_VALUE) return 0;
- /* check for valid fake dll file */
+ if (!ReadFile( *h, file_buffer, sizeof(file_buffer), &bytes_read, NULL ))
+ return -1;
+ if (bytes_read < sizeof(dos)) return -1;
- if (st.st_size < min_size) goto done;
- header_size = min( st.st_size, 4096 );
- if (read( fd, file_buffer, header_size ) != header_size) goto done;
- dos = file_buffer;
- if (dos->e_magic != IMAGE_DOS_SIGNATURE) goto done;
- if (dos->e_lfanew < sizeof(*dos) + 32) goto done;
+ dos = (IMAGE_DOS_HEADER *)file_buffer;
+ if (dos->e_magic != IMAGE_DOS_SIGNATURE) return -1;
+ if (dos->e_lfanew < sizeof(dos) + 32) return -1;
if (memcmp( dos + 1, builtin_signature, strlen(builtin_signature) + 1 ) &&
- memcmp( dos + 1, fakedll_signature, strlen(fakedll_signature) + 1 )) goto done;
- if (dos->e_lfanew +
FIELD_OFFSET(IMAGE_NT_HEADERS,OptionalHeader.MajorLinkerVersion) > header_size)
- goto done;
- nt = (IMAGE_NT_HEADERS *)((char *)file_buffer + dos->e_lfanew);
+ memcmp( dos + 1, fakedll_signature, strlen(fakedll_signature) + 1 )) return -1;
+ if (dos->e_lfanew +
FIELD_OFFSET(IMAGE_NT_HEADERS,OptionalHeader.MajorLinkerVersion) > bytes_read)
+ return -1;
+ nt = (IMAGE_NT_HEADERS *)(file_buffer + dos->e_lfanew);
if (nt->Signature == IMAGE_NT_SIGNATURE && nt->OptionalHeader.Magic !=
IMAGE_NT_OPTIONAL_HDR_MAGIC)
- {
/* wrong 32/64 type, pretend it doesn't exist */
- ret = 0;
- goto done;
- }
- if (st.st_size == header_size ||
- read( fd, (char *)file_buffer + header_size,
- st.st_size - header_size ) == st.st_size - header_size)
+ return 0;
+ return 1;
+}
+
+static BOOL write_fake_dll( const WCHAR *src_name, HANDLE src, HANDLE dest )
+{
+ BOOL ret = FALSE;
+
+ if (lstrlenW(src_name) > 2 && !wcscmp( src_name + lstrlenW(src_name) - 2,
L"16" ))
{
- *data = file_buffer;
- if (lstrlenW(name) > 2 && !wcscmp( name + lstrlenW(name) - 2,
L"16" ))
- extract_16bit_image( nt, data, size );
- ret = 1;
+ LARGE_INTEGER file_size;
+ HANDLE src_map;
+ if (!GetFileSizeEx( src, &file_size )) return FALSE;
+ src_map = CreateFileMappingA( src, NULL, PAGE_WRITECOPY, 0, 0, NULL );
+ if (src_map)
+ {
+ void *src_view = MapViewOfFile( src_map, FILE_MAP_COPY, 0, 0, 0 );
+ if (src_view)
+ {
+ DWORD bytes_written;
+ void *data = src_view;
+ SIZE_T size = file_size.QuadPart;
+ extract_16bit_image( &data, &size );
+ ret = WriteFile( dest, data, size, &bytes_written, NULL ) &&
bytes_written == size;
+ UnmapViewOfFile( src_view );
+ }
+ CloseHandle( src_map );
+ }
+ }
+ else
+ {
+ /* can't call CopyFileW here because files are open with no share mode */
+ /* wine extension: IOCTL_COPYCHUNK works even on local files */
+ SRV_COPYCHUNK_COPY copychunk = {
+ .ChunkCount = 1,
+ .Chunk = { [0] = { .Length = 1UL<<30 } }
+ };
+ SRV_COPYCHUNK_RESPONSE copychunk_resp;
+ DWORD count;
+ ret = DeviceIoControl( src, IOCTL_PREPARE_COPYCHUNK, NULL, 0,
©chunk.SourceFile,
+ sizeof(copychunk.SourceFile), &count, NULL );
+ if (!ret) return FALSE;
+ do {
+ ret = DeviceIoControl( dest, IOCTL_COPYCHUNK, ©chunk,
sizeof(copychunk),
+ ©chunk_resp, sizeof(copychunk_resp),
&count, NULL );
+ copychunk.Chunk[0].SourceOffset.QuadPart +=
copychunk_resp.TotalBytesWritten;
+ copychunk.Chunk[0].DestinationOffset.QuadPart +=
copychunk_resp.TotalBytesWritten;
+ } while (ret && copychunk_resp.TotalBytesWritten);
}
-done:
- close( fd );
return ret;
}
@@ -415,12 +434,11 @@ static const WCHAR *enum_load_path( unsigned int idx )
}
/* try to load a pre-compiled fake dll */
-static void *load_fake_dll( const WCHAR *name, SIZE_T *size )
+static int load_fake_dll( const WCHAR *name, HANDLE *h )
{
const WCHAR *build_dir = _wgetenv( L"WINEBUILDDIR" );
const WCHAR *path;
WCHAR *file, *ptr;
- void *data = NULL;
unsigned int i, pos, len, namelen, maxlen = 0;
WCHAR *p;
int res = 0;
@@ -433,7 +451,7 @@ static void *load_fake_dll( const WCHAR *name, SIZE_T *size )
while ((path = enum_load_path( i++ ))) maxlen = max( maxlen, lstrlenW(path) );
maxlen += ARRAY_SIZE(pe_dir) + len + 1;
- if (!(file = HeapAlloc( GetProcessHeap(), 0, maxlen * sizeof(WCHAR) ))) return NULL;
+ if (!(file = HeapAlloc( GetProcessHeap(), 0, maxlen * sizeof(WCHAR) ))) return -1;
pos = maxlen - len - 1;
lstrcpyW( file + pos, name );
@@ -449,7 +467,7 @@ static void *load_fake_dll( const WCHAR *name, SIZE_T *size )
ptr = prepend( ptr, ptr, namelen );
ptr = prepend( ptr, L"\\dlls", 5 );
ptr = prepend( ptr, build_dir, lstrlenW(build_dir) );
- if ((res = read_file( ptr, &data, size ))) goto done;
+ if ((res = read_file( ptr, h ))) goto done;
/* now as a program */
ptr = file + pos;
@@ -459,7 +477,7 @@ static void *load_fake_dll( const WCHAR *name, SIZE_T *size )
ptr = prepend( ptr, ptr, namelen );
ptr = prepend( ptr, L"\\programs", 9 );
ptr = prepend( ptr, build_dir, lstrlenW(build_dir) );
- if ((res = read_file( ptr, &data, size ))) goto done;
+ if ((res = read_file( ptr, h ))) goto done;
}
file[pos + len + 1] = 0;
@@ -467,15 +485,14 @@ static void *load_fake_dll( const WCHAR *name, SIZE_T *size )
{
ptr = prepend( file + pos, pe_dir, lstrlenW(pe_dir) );
ptr = prepend( ptr, path, lstrlenW(path) );
- if ((res = read_file( ptr, &data, size ))) break;
+ if ((res = read_file( ptr, h ))) break;
ptr = prepend( file + pos, path, lstrlenW(path) );
- if ((res = read_file( ptr, &data, size ))) break;
+ if ((res = read_file( ptr, h ))) break;
}
done:
HeapFree( GetProcessHeap(), 0, file );
- if (res == 1) return data;
- return NULL;
+ return res;
}
/* create the fake dll destination file */
@@ -505,7 +522,7 @@ static HANDLE create_dest_file( const WCHAR *name, BOOL delete )
{
if (GetLastError() == ERROR_PATH_NOT_FOUND) create_directories( name );
- h = CreateFileW( name, GENERIC_WRITE, 0, NULL, CREATE_NEW, 0, NULL );
+ h = CreateFileW( name, GENERIC_READ|GENERIC_WRITE, 0, NULL, CREATE_NEW, 0, NULL
);
if (h == INVALID_HANDLE_VALUE)
ERR( "failed to create %s (error=%u)\n", debugstr_w(name),
GetLastError() );
}
@@ -838,23 +855,30 @@ static BOOL CALLBACK register_resource( HMODULE module, LPCWSTR
type, LPWSTR nam
return TRUE;
}
-static void register_fake_dll( const WCHAR *name, const void *data, size_t size, struct
list *delay_copy )
+static void register_fake_dll( const WCHAR *name, HANDLE h, struct list *delay_copy )
{
const IMAGE_RESOURCE_DIRECTORY *resdir;
LDR_RESOURCE_INFO info;
HRESULT hr = S_OK;
- HMODULE module = (HMODULE)((ULONG_PTR)data | 1);
struct dll_data dll_data = { delay_copy, name, 0 };
WCHAR buffer[MAX_PATH];
const WCHAR *p;
+ HANDLE mapping;
+ HMODULE module;
if (!(p = wcsrchr( name, '\\' ))) p = name;
else p++;
dll_data.src_len = p - name;
+
+ mapping = CreateFileMappingA( h, NULL, PAGE_READONLY, 0, 0, NULL );
+ if (!mapping) return;
+ module = (HMODULE)((ULONG_PTR)MapViewOfFile( mapping, FILE_MAP_COPY, 0, 0, 0 ) | 1);
+ if (!module) goto out;
+
EnumResourceNamesW( module, (WCHAR*)RT_MANIFEST, register_manifest,
(LONG_PTR)&dll_data );
info.Type = (ULONG_PTR)L"WINE_REGISTRY";
- if (LdrFindResourceDirectory_U( module, &info, 1, &resdir )) return;
+ if (LdrFindResourceDirectory_U( module, &info, 1, &resdir )) goto out;
if (!registrar)
{
@@ -869,7 +893,7 @@ static void register_fake_dll( const WCHAR *name, const void *data,
size_t size,
if (!registrar)
{
ERR( "failed to create IRegistrar: %x\n", hr );
- return;
+ goto out;
}
}
@@ -880,22 +904,24 @@ static void register_fake_dll( const WCHAR *name, const void *data,
size_t size,
IRegistrar_AddReplacement( registrar, L"SystemRoot", buffer );
EnumResourceNamesW( module, L"WINE_REGISTRY", register_resource,
(LONG_PTR)&hr );
if (FAILED(hr)) ERR( "failed to register %s: %x\n", debugstr_w(name), hr
);
+out:
+ if (module) UnmapViewOfFile(module);
+ CloseHandle( mapping );
}
/* copy a fake dll file to the dest directory */
static int install_fake_dll( WCHAR *dest, WCHAR *file, BOOL delete, struct list
*delay_copy )
{
int ret;
- SIZE_T size;
- void *data;
- DWORD written;
WCHAR *destname = dest + lstrlenW(dest);
WCHAR *name = wcsrchr( file, '\\' ) + 1;
WCHAR *end = name + lstrlenW(name);
SIZE_T len = end - name;
+ HANDLE hsrc = INVALID_HANDLE_VALUE;
- if (!(ret = read_file( file, &data, &size )))
+ if (!(ret = read_file( file, &hsrc )))
{
+ if (hsrc != INVALID_HANDLE_VALUE) CloseHandle( hsrc );
*end = 0;
return 0;
}
@@ -913,13 +939,14 @@ static int install_fake_dll( WCHAR *dest, WCHAR *file, BOOL delete,
struct list
{
TRACE( "%s -> %s\n", debugstr_w(file), debugstr_w(dest) );
- ret = (WriteFile( h, data, size, &written, NULL ) && written ==
size);
+ ret = write_fake_dll( file, hsrc, h );
if (!ret) ERR( "failed to write to %s (error=%u)\n",
debugstr_w(dest), GetLastError() );
- CloseHandle( h );
- if (ret) register_fake_dll( dest, data, size, delay_copy );
+ if (ret) register_fake_dll( dest, h, delay_copy );
else DeleteFileW( dest );
+ CloseHandle( h );
}
}
+ if (hsrc != INVALID_HANDLE_VALUE) CloseHandle( hsrc );
*destname = 0; /* restore it for next file */
*end = 0;
return ret;
@@ -928,30 +955,25 @@ static int install_fake_dll( WCHAR *dest, WCHAR *file, BOOL delete,
struct list
static void delay_copy_files( struct list *delay_copy )
{
struct delay_copy *copy, *next;
- DWORD written;
- SIZE_T size;
- void *data;
- HANDLE h;
int ret;
LIST_FOR_EACH_ENTRY_SAFE( copy, next, delay_copy, struct delay_copy, entry )
{
+ HANDLE h, h2;
list_remove( ©->entry );
- ret = read_file( copy->src, &data, &size );
- if (ret != 1)
- {
- HeapFree( GetProcessHeap(), 0, copy );
- continue;
- }
+ ret = read_file( copy->src, &h2 );
+ if (ret != 1) goto next;
h = create_dest_file( copy->dest, FALSE );
if (h && h != INVALID_HANDLE_VALUE)
{
- ret = (WriteFile( h, data, size, &written, NULL ) && written ==
size);
+ ret = write_fake_dll( copy->src, h2, h );
if (!ret) ERR( "failed to write to %s (error=%u)\n",
debugstr_w(copy->dest), GetLastError() );
CloseHandle( h );
if (!ret) DeleteFileW( copy->dest );
}
+next:
+ if (h2 != INVALID_HANDLE_VALUE) CloseHandle( h2 );
HeapFree( GetProcessHeap(), 0, copy );
}
}
@@ -1041,11 +1063,9 @@ static BOOL create_wildcard_dlls( const WCHAR *dirname, const WCHAR
*wildcard, B
BOOL create_fake_dll( const WCHAR *name, const WCHAR *source )
{
struct list delay_copy = LIST_INIT( delay_copy );
- HANDLE h;
+ HANDLE h, h2;
BOOL ret;
- SIZE_T size;
const WCHAR *filename;
- void *buffer;
BOOL delete = !wcscmp( source, L"-" ); /* '-' source means delete
the file */
if (!(filename = wcsrchr( name, '\\' ))) filename = name;
@@ -1064,12 +1084,10 @@ BOOL create_fake_dll( const WCHAR *name, const WCHAR *source )
if (!(h = create_dest_file( name, delete ))) return TRUE; /* not a fake dll */
if (h == INVALID_HANDLE_VALUE) return FALSE;
- if ((buffer = load_fake_dll( source, &size )))
+ if (load_fake_dll( source, &h2 ))
{
- DWORD written;
-
- ret = (WriteFile( h, buffer, size, &written, NULL ) && written ==
size);
- if (ret) register_fake_dll( name, buffer, size, &delay_copy );
+ ret = write_fake_dll( source, h2, h );
+ if (ret) register_fake_dll( name, h, &delay_copy );
else ERR( "failed to write to %s (error=%u)\n", debugstr_w(name),
GetLastError() );
}
else
@@ -1078,7 +1096,9 @@ BOOL create_fake_dll( const WCHAR *name, const WCHAR *source )
ret = build_fake_dll( h, name );
}
+ if (h2 != INVALID_HANDLE_VALUE) CloseHandle( h2 );
CloseHandle( h );
+
if (!ret) DeleteFileW( name );
delay_copy_files( &delay_copy );
@@ -1091,8 +1111,6 @@ BOOL create_fake_dll( const WCHAR *name, const WCHAR *source )
*/
void cleanup_fake_dlls(void)
{
- if (file_buffer) VirtualFree( file_buffer, 0, MEM_RELEASE );
- file_buffer = NULL;
HeapFree( GetProcessHeap(), 0, handled_dlls );
handled_dlls = NULL;
handled_count = handled_total = 0;
--
2.35.0
9:50 p.m.
New subject: [PATCH v3 4/5] lmshare.h: define STYPE_TEMPORARY
---
include/lmshare.h | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/include/lmshare.h b/include/lmshare.h
index 442cf5ef306..afec40542f4 100644
--- a/include/lmshare.h
+++ b/include/lmshare.h
@@ -68,11 +68,12 @@ NET_API_STATUS WINAPI
NetShareEnumSticky(LMSTR,DWORD,LPBYTE*,DWORD,LPDWORD,LPDWO
NET_API_STATUS WINAPI NetShareGetInfo(LMSTR,LMSTR,DWORD,LPBYTE*);
NET_API_STATUS WINAPI NetShareSetInfo(LMSTR,LMSTR,DWORD,LPBYTE,LPDWORD);
-#define STYPE_DISKTREE 0
-#define STYPE_PRINTQ 1
-#define STYPE_DEVICE 2
-#define STYPE_IPC 3
-#define STYPE_SPECIAL 0x80000000
+#define STYPE_DISKTREE 0
+#define STYPE_PRINTQ 1
+#define STYPE_DEVICE 2
+#define STYPE_IPC 3
+#define STYPE_TEMPORARY 0x40000000
+#define STYPE_SPECIAL 0x80000000
NET_API_STATUS WINAPI NetSessionDel(LMSTR,LMSTR,LMSTR);
NET_API_STATUS WINAPI
NetSessionEnum(LMSTR,LMSTR,LMSTR,DWORD,LPBYTE*,DWORD,LPDWORD,LPDWORD,LPDWORD);
--
2.35.0
9:50 p.m.
New subject: [PATCH v3 5/5] kernel32/tests: add IOCTL_COPYCHUNK test
---
dlls/kernel32/tests/file.c | 137 +++++++++++++++++++++++++++++++++++++
1 file changed, 137 insertions(+)
diff --git a/dlls/kernel32/tests/file.c b/dlls/kernel32/tests/file.c
index 60fe532adc2..ba4839231c7 100644
--- a/dlls/kernel32/tests/file.c
+++ b/dlls/kernel32/tests/file.c
@@ -31,9 +31,13 @@
#include "windef.h"
#include "winbase.h"
#include "winerror.h"
+#include "winioctl.h"
#include "winternl.h"
#include "winnls.h"
#include "fileapi.h"
+#include "lmaccess.h"
+#include "lmshare.h"
+#include "lmerr.h"
#undef DeleteFile /* needed for FILE_DISPOSITION_INFO */
@@ -59,6 +63,7 @@ static void (WINAPI *pRtlInitAnsiString)(PANSI_STRING,PCSZ);
static void (WINAPI *pRtlFreeUnicodeString)(PUNICODE_STRING);
static BOOL (WINAPI *pSetFileCompletionNotificationModes)(HANDLE, UCHAR);
static HANDLE (WINAPI *pFindFirstStreamW)(LPCWSTR filename, STREAM_INFO_LEVELS infolevel,
void *data, DWORD flags);
+static const char * (CDECL *pwine_get_version)(void);
static char filename[MAX_PATH];
static const char sillytext[] =
@@ -108,6 +113,7 @@ static void InitFunctionPointers(void)
pReOpenFile = (void *) GetProcAddress(hkernel32, "ReOpenFile");
pSetFileCompletionNotificationModes = (void *)GetProcAddress(hkernel32,
"SetFileCompletionNotificationModes");
pFindFirstStreamW = (void *)GetProcAddress(hkernel32, "FindFirstStreamW");
+ pwine_get_version = (void *)GetProcAddress(hntdll, "wine_get_version");
}
static void create_file( const char *path )
@@ -6130,6 +6136,136 @@ static void test_eof(void)
ok(ret, "failed to delete %s, error %u\n", debugstr_a(filename),
GetLastError());
}
+static void test_copychunk(void)
+{
+ static const char testString[] = "hello world";
+ char testBuf[sizeof(testString)-1];
+ WCHAR temp_path[MAX_PATH];
+ WCHAR temp_file[MAX_PATH];
+ WCHAR share_name[MAX_PATH];
+ WCHAR share_path[MAX_PATH];
+ HANDLE src, dest;
+ int ret;
+ DWORD written, count;
+ FILE_END_OF_FILE_INFO info;
+ SRV_COPYCHUNK_COPY copychunk;
+ SRV_COPYCHUNK_RESPONSE copychunk_resp;
+ SRV_REQUEST_RESUME_KEY key;
+ NET_API_STATUS (NET_API_FUNCTION *pNetShareAdd)(LMSTR, DWORD, LPBYTE, LPDWORD);
+ NET_API_STATUS (NET_API_FUNCTION *pNetShareDel)(LMSTR, LMSTR, DWORD);
+ GetTempPathW(MAX_PATH, temp_path);
+
+ if (pwine_get_version == NULL)
+ {
+ HMODULE hnetapi32;
+ SHARE_INFO_2 shi2 = {
+ .shi2_type = STYPE_DISKTREE | STYPE_TEMPORARY | STYPE_SPECIAL,
+ .shi2_permissions = ACCESS_ALL,
+ .shi2_max_uses = -1,
+ };
+ hnetapi32 = LoadLibraryA("netapi32");
+ if (hnetapi32 == NULL)
+ {
+ win_skip("could not find hnetapi32\n");
+ return;
+ }
+ pNetShareAdd = (void *)GetProcAddress(hnetapi32, "NetShareAdd");
+ if (pNetShareAdd == NULL)
+ {
+ win_skip("could not find NetShareAdd\n");
+ return;
+ }
+ pNetShareDel = (void *)GetProcAddress(hnetapi32, "NetShareDel");
+ if (pNetShareDel == NULL)
+ {
+ win_skip("could not find NetShareDel\n");
+ return;
+ }
+ srand((unsigned)time(NULL));
+ do {
+ int r = rand();
+ swprintf(share_path, MAX_PATH, L"%lswinetest%d", temp_path, r);
+ swprintf(share_name, MAX_PATH, L"winetest%d$", r);
+ ret = CreateDirectoryW(share_path, NULL);
+ if (ret == 0 && GetLastError() != ERROR_ALREADY_EXISTS)
+ {
+ ok(ret != 0, "CreateDirectoryW error %d\n", GetLastError());
+ return;
+ }
+ } while (ret == 0);
+ shi2.shi2_netname = share_name;
+ shi2.shi2_path = share_path;
+ ret = pNetShareAdd(NULL, 2, (BYTE *)&shi2, NULL);
+ if (ret == ERROR_ACCESS_DENIED)
+ {
+ win_skip("windows IOCTL_COPYCHUNK needs admin for NetShareAdd\n");
+ RemoveDirectoryW(share_path);
+ return;
+ }
+ ok(ret == NERR_Success || ret == ERROR_ACCESS_DENIED, "NetShareAdd error
%d\n", ret);
+ swprintf(temp_file, MAX_PATH, L"\\\\localhost\\%ls\\file",
share_name);
+ }
+ else
+ {
+ ret = GetTempFileNameW(temp_path, L"tmp", 0, temp_file);
+ ok(ret != 0, "GetTempFileNameA error %d\n", GetLastError());
+ }
+
+ src = CreateFileW(temp_file, GENERIC_READ | GENERIC_WRITE, 0, NULL,
+ OPEN_ALWAYS, FILE_FLAG_DELETE_ON_CLOSE, NULL);
+ ok(src != INVALID_HANDLE_VALUE, "CreateFileW error %d\n", GetLastError());
+
+ ret = WriteFile(src, testString, sizeof(testString) - 1, &written, NULL);
+ ok(ret != 0, "WriteFile error %d\n", GetLastError());
+ ok(written == sizeof(testString) - 1, "short write to regular file\n");
+
+ ret = FlushFileBuffers(src);
+ ok(ret != 0, "FlushFileBuffers error %d\n", GetLastError());
+
+ wcscat(temp_file, L"2");
+ dest = CreateFileW(temp_file, GENERIC_READ | GENERIC_WRITE, 0, NULL,
+ CREATE_NEW, FILE_FLAG_DELETE_ON_CLOSE, NULL);
+ ok(dest != INVALID_HANDLE_VALUE, "CreateFileW error %d\n",
GetLastError());
+
+ info.EndOfFile.QuadPart = written;
+ ret = SetFileInformationByHandle(dest, FileEndOfFileInfo, &info, sizeof(info));
+ ok(ret != 0, "set EOF failed: %d\n", GetLastError());
+
+ ret = DeviceIoControl( src, IOCTL_PREPARE_COPYCHUNK, NULL, 0, &key,
+ sizeof(key), &count, NULL );
+ ok(ret != 0, "IOCTL_PREPARE_COPYCHUNK error %d\n", GetLastError());
+ ok(count != 0, "IOCTL_PREPARE_COPYCHUNK wrong return size: %d\n", count);
+
+ copychunk.SourceFile = key.Key;
+ copychunk.ChunkCount = 1;
+ copychunk.Chunk[0].Length = written;
+ copychunk.Chunk[0].SourceOffset.QuadPart = 0;
+ copychunk.Chunk[0].DestinationOffset.QuadPart = 0;
+ ret = DeviceIoControl( dest, IOCTL_COPYCHUNK, ©chunk, sizeof(copychunk),
+ ©chunk_resp, sizeof(copychunk_resp), &count, NULL
);
+ ok(ret != 0, "IOCTL_COPYCHUNK error %d\n", GetLastError());
+ ok(count == sizeof(copychunk_resp), "IOCTL_COPYCHUNK wrong return size:
%d\n", count);
+ ok(copychunk_resp.ChunksWritten == 1,
+ "wrong chunks count: %d, should be 1\n", copychunk_resp.ChunksWritten);
+ ok(copychunk_resp.ChunkBytesWritten == 0,
+ "wrong chunk bytes written: %d, should be 0\n",
copychunk_resp.ChunkBytesWritten);
+ ok(copychunk_resp.TotalBytesWritten == written,
+ "wrong total bytes written: %d, should be %d\n",
copychunk_resp.TotalBytesWritten, written);
+
+ ret = ReadFile(dest, testBuf, sizeof(testBuf), &count, NULL);
+ ok(ret != 0, "ReadFile error %d\n", GetLastError());
+ ok(count == sizeof(testBuf), "short read from regular file\n");
+ ok(!memcmp(testString, testBuf, sizeof(testBuf)), "copied contents not identical
to original\n");
+
+ CloseHandle(dest);
+ CloseHandle(src);
+ if (pwine_get_version == NULL)
+ {
+ pNetShareDel(NULL, share_name, 0);
+ RemoveDirectoryW(share_path);
+ }
+}
+
START_TEST(file)
{
char temp_path[MAX_PATH];
@@ -6208,4 +6344,5 @@ START_TEST(file)
test_hard_link();
test_move_file();
test_eof();
+ test_copychunk();
}
--
2.35.0
10:24 p.m.
Hello,
first, thanks for working on this, would be a welcome improvement!
Do I understand that right that it wouldn't work for ext4 and there's not
really anything we could do to support it?
Regards,
Fabian Maurer
On Donnerstag, 27. Januar 2022 22:50:51 CET Alex Xu (Hello71) wrote:
Over the years, Wine prefixes have gotten bigger and
bigger, for a
number of reasons. Creating a new Wine prefix for each application is
still the current recommendation, as despite the best efforts of Wine
developers, some applications still require system-wide workarounds.
This leads to significant bloat for each application installed. With a
MinGW build of Wine without Mono or Gecko, new 32-bit prefixes are over
150 MB, and new 64-bit prefixes are over 300 MB. The vast majority of
these files are byte-for-byte identical to Wine's central DLL copies.
This patch set implements reflink support in Wine via the
copy_file_range syscall. The reasons for selecting copy_file_range over
FICLONE are outlined in patch 2. A previous unpublished version of this
patch set used FICLONERANGE, but it was less convenient to use from
setupapi and has inferior system support.
When reflink is supported by the underlying filesystem, new Wine prefix
sizes with Mono and Gecko disabled are reduced to less than 1 MB. The
resulting Wine prefix is byte-for-byte identical to one created without
reflink, but occupies less space on disk. If hard links or symlinks were
used, if an application such as winetricks writes to a system file, it
would overwrite the central copy. With reflink, the file blocks will be
transparently copied by the Linux kernel so that each Wine prefix can be
independent.
Some files cannot be deduplicated in the current Wine system, as they
are dynamically generated during the Wine prefix installation process.
These include 16-bit fake DLLs and manifest files. In theory, it should
be possible to pre-generate these files, but considering the Wine prefix
size is already reduced to less than 1 MB, the extra space savings are
likely not worth the effort.
v1->v2: Fix/improve errno handling.
v2->v3: Rebase.
Alex Xu (Hello71) (5):
ntdll: add support for IOCTL_COPYCHUNK.
kernelbase: use IOCTL_COPYCHUNK in CopyFile*
setupapi: Use IOCTL_COPYCHUNK, avoid buffering whole file
lmshare.h: define STYPE_TEMPORARY
kernel32/tests: add IOCTL_COPYCHUNK test
configure | 6 ++
configure.ac | 1 +
dlls/kernel32/tests/file.c | 137 +++++++++++++++++++++++++
dlls/kernelbase/file.c | 44 ++++-----
dlls/ntdll/unix/file.c | 121 +++++++++++++++++++++++
dlls/setupapi/fakedll.c | 198 ++++++++++++++++++++-----------------
include/config.h.in | 3 +
include/lmshare.h | 11 ++-
include/winioctl.h | 34 +++++++
9 files changed, 434 insertions(+), 121 deletions(-)
5 Feb
5 Feb
4:51 p.m.
Excerpts from Fabian Maurer's message of January 27, 2022 5:24 pm:
Hello,
first, thanks for working on this, would be a welcome improvement!
Do I understand that right that it wouldn't work for ext4 and there's not
really anything we could do to support it?
Regards,
Fabian Maurer
ext4 does not support reflink, so copy_file_range cannot reduce disk
space usage. However, it will still allow the kernel to do the file copy
itself, avoiding copying the data to userspace.
Full reflink functionality with arbitrary offsets and overwrite is
non-trivial to implement in the filesystem. However, we really only need
the limited functionality of a whole-file reflink. This could be
simulated using a hard or symbolic link which gets broken by Wine when
any write occurs. However, as with reparse points, this requires
modifying every file access that Wine makes, which would be a
significant undertaking. Additionally, as explained in the cover letter,
this would potentially break Unix-side programs modifying the Wine
prefix, such as winetricks.
Alex.
6 Feb
6 Feb
2:51 p.m.
"Alex Xu (Hello71)" <alex_y_xu(a)yahoo.ca> writes:
Excerpts from Fabian Maurer's message of January
27, 2022 5:24 pm:
I'm not convinced that it would be that hard, and if it can work it
would clearly be a better approach. Prefix size is a real problem, but I
don't think reflinks are the answer.
--
Alexandre Julliard
julliard(a)winehq.org
Hello,
first, thanks for working on this, would be a welcome improvement!
Do I understand that right that it wouldn't work for ext4 and there's not
really anything we could do to support it?
Regards,
Fabian Maurer
ext4 does not support reflink, so copy_file_range cannot reduce disk
space usage. However, it will still allow the kernel to do the file copy
itself, avoiding copying the data to userspace.
Full reflink functionality with arbitrary offsets and overwrite is
non-trivial to implement in the filesystem. However, we really only need
the limited functionality of a whole-file reflink. This could be
simulated using a hard or symbolic link which gets broken by Wine when
any write occurs. However, as with reparse points, this requires
modifying every file access that Wine makes, which would be a
significant undertaking.
7 Feb
7 Feb
7:44 p.m.
Excerpts from Alexandre Julliard's message of February 6, 2022 9:51 am:
"Alex Xu (Hello71)"
<alex_y_xu(a)yahoo.ca> writes:
I disfavor the "magic symlink" approach for three reasons:
1. Implementing copy_file_range for CopyFile improves performance not
only for wineboot, but also for every program using CopyFile. It allows
reflink but also remote-side copy for cifs on Linux. I doubt anybody has
Wine prefix on cifs, but maybe some people put files or programs on a
network drive.
2. Who will review the resulting patch? This submission is now the third
revision of this patchset with no non-trivial review. While we may
disagree on whether "magic symlink" will be 2x or 10x the current
patchset, it will certainly be larger. In particular, the setupapi part,
which is the most complex part of this patchset, will certainly be
larger and even more complex; passing file names down to write_fake_dll
is harder than passing file handles, not easier.
3. Who will fix the resulting breakage in winetricks, protontricks, and
who knows what other programs? As far as I can tell, any option for
representing the magic link on the Unix side will result in issues.
Either it uses symlinks, and "cp override_mydll.dll .../mydll.dll" will
be broken, or it doesn't create the Unix side files, so ls will be
broken, or it uses empty files, which is possibly the least worst
solution but will break either "cp mydll.dll mydll2.dll" or "cp -a
$WINEPREFIX $WINEPREFIX.bak", depending on the Wine implementation.
That being said, I am perfectly happy for someone else to write this
"magic symlink" implementation. I agree that the full reflink
functionality is not required, and it would be more beneficial to users
to reduce prefix sizes on ext4. Even though most other Linux filesystems
support reflink (even XFS, which is not CoW), ext4 is still the most
popular option. However, I will not implement this, as I have already
spent significant time working on this with no serious response from
Wine project, only some side comments. "can it work on mac" is a fine
question to ask, but is not actionable.
Alex.
Full reflink functionality with arbitrary offsets
and overwrite is
non-trivial to implement in the filesystem. However, we really only need
the limited functionality of a whole-file reflink. This could be
simulated using a hard or symbolic link which gets broken by Wine when
any write occurs. However, as with reparse points, this requires
modifying every file access that Wine makes, which would be a
significant undertaking.
I'm not convinced that it would be that hard, and if it can work it
would clearly be a better approach. Prefix size is a real problem, but I
don't think reflinks are the answer.
--
Alexandre Julliard
julliard(a)winehq.org
8 Feb
8 Feb
8:15 p.m.
On Mon, Feb 7, 2022 at 12:44 PM Alex Xu (Hello71) <alex_y_xu(a)yahoo.ca> wrote:
Excerpts from Alexandre Julliard's message of February 6, 2022 9:51 am:
I'm coming late to the party here (been on a work trip for about a
month), but are you talking about the proposal to have the core Wine
DLLs use special symlinks inside the prefix that are copy-on-write?
"Alex Xu (Hello71)"
<alex_y_xu(a)yahoo.ca> writes:
I disfavor the "magic symlink" approach for three reasons: Full reflink functionality with arbitrary offsets
and overwrite is
non-trivial to implement in the filesystem. However, we really only need
the limited functionality of a whole-file reflink. This could be
simulated using a hard or symbolic link which gets broken by Wine when
any write occurs. However, as with reparse points, this requires
modifying every file access that Wine makes, which would be a
significant undertaking.
I'm not convinced that it would be that hard, and if it can work it
would clearly be a better approach. Prefix size is a real problem, but I
don't think reflinks are the answer.
--
Alexandre Julliard
julliard(a)winehq.org
1. Implementing copy_file_range for CopyFile improves
performance not
only for wineboot, but also for every program using CopyFile. It allows
reflink but also remote-side copy for cifs on Linux. I doubt anybody has
Wine prefix on cifs, but maybe some people put files or programs on a
network drive.
We could conceivably use the symlink approach for this (as long as
you're working with whole files).
2. Who will review the resulting patch? This
submission is now the third
revision of this patchset with no non-trivial review.
This mailing list and, ultimately, AJ. My interpretation of the
response you have is that he's not convinced that implementing full
reflink support is as hard as you think. You might ask him for
clarification, as it seems to me that depending on what you're talking
about that that would be very difficult.
While we may
disagree on whether "magic symlink" will be 2x or 10x the current
patchset, it will certainly be larger. In particular, the setupapi part,
which is the most complex part of this patchset, will certainly be
larger and even more complex; passing file names down to write_fake_dll
is harder than passing file handles, not easier.
This comment is why I'm wondering if you're talking about something
else. My view of the symlink approach is that (after some comments
from an RFC) there's about three steps:
1) implement generic reparse point storage (convert reparse points
into symlinks with contents that are not usable outside Wine) and
retrieval
2) implement specialized reparse point storage (convert common reparse
points into unix-usable paths) and retrieval
3) implement a Wine-specific reparse point with copy-on-write behavior
Steps 1&2 of this process are useful in general (since we don't
currently have reparse point support), which makes step #3 a much
smaller lift once the rest of the infrastructure (1&2) are in place.
I will grant you that the full implementation is a lot of patches (I
have not separated 1&2 yet, and currently have 39 patches just for
that), but a prototype I put together to test the feasibility of #3 is
only about 3 small patches.
3. Who will fix the resulting breakage in winetricks,
protontricks, and
who knows what other programs?
The people that maintain these applications will be forced to update
if things break outside of Wine.
As far as I can tell, any option for
representing the magic link on the Unix side will result in issues.
Either it uses symlinks, and "cp override_mydll.dll .../mydll.dll" will
be broken, or it doesn't create the Unix side files, so ls will be
broken, or it uses empty files, which is possibly the least worst
solution but will break either "cp mydll.dll mydll2.dll" or "cp -a
$WINEPREFIX $WINEPREFIX.bak", depending on the Wine implementation.
That really depends upon what you stick into the symlink and whether
you expect the symlink to work outside Wine. Part of the feedback I
got from my last symlink/reparse point RFC was "does this really need
to work outside of Wine?" I think it's important to have standard
Unix tools work with any Wine symlinks, but if the contents of the
symlink look something like "${WINEPREFIX}/path/to/target" then that
makes it possible to backup (cp -a) a prefix where a "proper"
implementation that has fully working links on the Unix side would not
allow that. For what you're doing here I would expect that you would
want to use absolute paths or prefix-relative paths, which would not
break if you use "cp -a override_mydll.dll .../mydll.dll" (though you
would need to use "-a").
That being said, I am perfectly happy for someone else
to write this
"magic symlink" implementation.
I've been putting something together in that vein since 2014, though I
am taking a very different approach from your proposal. People might
like your approach better.
I agree that the full reflink
functionality is not required, and it would be more beneficial to users
to reduce prefix sizes on ext4.
Symlinks work cross-volume, so if you have /usr/lib/wine on a
different volume (very common in a lot of environments) then reflinks
won't save you.
Even though most other Linux filesystems
support reflink (even XFS, which is not CoW), ext4 is still the most
popular option. However, I will not implement this, as I have already
spent significant time working on this with no serious response from
Wine project, only some side comments. "can it work on mac" is a fine
question to ask, but is not actionable.
Wine doesn't particularly like to add features that are not
generalizable, as these features have a tendency to add a large
maintenance burden and don't benefit the vast majority of users. An
approach that sometimes works here is to create a general solution
(possibly symlinks) that works for everyone that with a simple
extension (reflinks) then works _better_ if that feature is available.
This approach also means that when there's a bug that you can ask
people "have you tried turning off feature X?" to narrow in on what's
wrong.
Best,
Erich
19 Feb
19 Feb
5:14 p.m.
Excerpts from Erich E. Hoover's message of February 8, 2022 3:15 pm:
My point does not relate to a nominal authority, but a practical matter.
There is no point in undertaking this work for the result to be ignored
by "this mailing list and, ultimately, AJ", as has been the case with
this patchset.
As I have stated, I would be happy to see an alternative, "cleaner" or
"more portable" implementation. However, reparse points have obvious
downsides, including not being usable outside of Wine (unlike reflinks,
which behave as regular files), and more importantly, are not
implemented yet and have no clear timeline for being implemented. This
patchset is implemented now and works now.
While I don't speak for AJ and other Wine developers, I believe the
objective of Wine is to allow unmodified and minimally modified
(Winelib) programs to work on Linux and other Unix-like operating
systems. It is highly user-unfriendly to break almost all Wine prefix
management programs and paradigms in the pursuit of a
theoretically-cleaner but practically less-usable solution.
Yes, there are various options, all of which have drawbacks. Reflinks
have the drawback of not working on a currently-popular filesystem, but
where supported, have absolutely no user-visible issues.
Additionally, neither $, {, nor } are forbidden characters in Windows
filenames, so special-casing names starting with ${WINEPREFIX} would
lead to potential collisions.
This is a better argument. However, I believe that a majority of Wine
users have /usr/lib/wine on the same filesystem as their prefixes. The
exceptions are primarily poorly informed "sysadmins" with split /usr for
no good reason, those using LVM as a crappy quota implementation, and
those using experimental immutable /usr systems. In my view, the first
group can be safely ignored, the second group is mostly server
administrators and unlikely to be running Wine, and the third group is
currently extremely small and also unlikely to be running Wine, or if
they are, unlikely to be using multiple prefixes.
Your proposal of "a general solution, with an implementation-specific
optimization" is exactly the point of copy_file_range. It offloads the
burden of determining which optimizations are available and the precise
implementation to the kernel, which knows the underlying filesystem and
its supported methods.
I don't see IOCTL_COPYCHUNK as "adding a large maintenance burden". The
net added code is barely more than 100 LoC excluding defines and tests.
This is certainly far less than even the most basic reparse point
implementation. Additionally, contrary to your claim, it benefits the
substantial portion of users on btrfs, xfs, and in the future, bcachefs.
Also, as I have explained, it benefits not only Wine prefix copies, but
also file copies inside Wine.
What issues do you foresee with copy_file_range that could justify
adding a switch (with the associated maintenance burden of *that*) to
disable it?
On Mon, Feb 7, 2022 at 12:44 PM Alex Xu (Hello71)
<alex_y_xu(a)yahoo.ca> wrote:
I think it is a very bad idea to have CopyFile create symlinks on the
Unix side for user files. Magic symlinks should only be used for
internal files which are assumed to always be present, otherwise massive
confusion will result. Specifically, copying one directory to another in
Wine and then deleting the source directory outside of Wine must *not*
result in complete data loss.
I disfavor the "magic symlink" approach
for three reasons:
I'm coming late to the party here (been on a work trip for about a
month), but are you talking about the proposal to have the core Wine
DLLs use special symlinks inside the prefix that are copy-on-write?
1. Implementing copy_file_range for CopyFile
improves performance not
only for wineboot, but also for every program using CopyFile. It allows
reflink but also remote-side copy for cifs on Linux. I doubt anybody has
Wine prefix on cifs, but maybe some people put files or programs on a
network drive.
We could conceivably use the symlink approach for this (as long as
you're working with whole files). 2. Who will
review the resulting patch? This submission is now the third
revision of this patchset with no non-trivial review.
This mailing list and, ultimately, AJ. My interpretation of the
response you have is that he's not convinced that implementing full
reflink support is as hard as you think. You might ask him for
clarification, as it seems to me that depending on what you're talking
about that that would be very difficult. While we may
disagree on whether "magic symlink" will be 2x or 10x the current
patchset, it will certainly be larger. In particular, the setupapi part,
which is the most complex part of this patchset, will certainly be
larger and even more complex; passing file names down to write_fake_dll
is harder than passing file handles, not easier.
This comment is why I'm wondering if you're talking about something
else. My view of the symlink approach is that (after some comments
from an RFC) there's about three steps:
1) implement generic reparse point storage (convert reparse points
into symlinks with contents that are not usable outside Wine) and
retrieval
2) implement specialized reparse point storage (convert common reparse
points into unix-usable paths) and retrieval
3) implement a Wine-specific reparse point with copy-on-write behavior
Steps 1&2 of this process are useful in general (since we don't
currently have reparse point support), which makes step #3 a much
smaller lift once the rest of the infrastructure (1&2) are in place.
I will grant you that the full implementation is a lot of patches (I
have not separated 1&2 yet, and currently have 39 patches just for
that), but a prototype I put together to test the feasibility of #3 is
only about 3 small patches. 3. Who will
fix the resulting breakage in winetricks, protontricks, and
who knows what other programs?
The people that maintain these applications will be forced to update
if things break outside of Wine. As far as I
can tell, any option for
representing the magic link on the Unix side will result in issues.
Either it uses symlinks, and "cp override_mydll.dll .../mydll.dll" will
be broken, or it doesn't create the Unix side files, so ls will be
broken, or it uses empty files, which is possibly the least worst
solution but will break either "cp mydll.dll mydll2.dll" or "cp -a
$WINEPREFIX $WINEPREFIX.bak", depending on the Wine implementation.
That really depends upon what you stick into the symlink and whether
you expect the symlink to work outside Wine. Part of the feedback I
got from my last symlink/reparse point RFC was "does this really need
to work outside of Wine?" I think it's important to have standard
Unix tools work with any Wine symlinks, but if the contents of the
symlink look something like "${WINEPREFIX}/path/to/target" then that
makes it possible to backup (cp -a) a prefix where a "proper"
implementation that has fully working links on the Unix side would not
allow that. For what you're doing here I would expect that you would
want to use absolute paths or prefix-relative paths, which would not
break if you use "cp -a override_mydll.dll .../mydll.dll" (though you
would need to use "-a"). That being
said, I am perfectly happy for someone else to write this
"magic symlink" implementation.
I've been putting something together in that vein since 2014, though I
am taking a very different approach from your proposal. People might
like your approach better.
I agree that the full reflink
functionality is not required, and it would be more beneficial to users
to reduce prefix sizes on ext4.
Symlinks work cross-volume, so if you have /usr/lib/wine on a
different volume (very common in a lot of environments) then reflinks
won't save you. Even though
most other Linux filesystems
support reflink (even XFS, which is not CoW), ext4 is still the most
popular option. However, I will not implement this, as I have already
spent significant time working on this with no serious response from
Wine project, only some side comments. "can it work on mac" is a fine
question to ask, but is not actionable.
Wine doesn't particularly like to add features that are not
generalizable, as these features have a tendency to add a large
maintenance burden and don't benefit the vast majority of users. An
approach that sometimes works here is to create a general solution
(possibly symlinks) that works for everyone that with a simple
extension (reflinks) then works _better_ if that feature is available.
This approach also means that when there's a bug that you can ask
people "have you tried turning off feature X?" to narrow in on what's
wrong. Best,
Erich
Alex.
804
days inactive
827
days old
11 comments
4 participants
participants (4)
-
Alex Xu (Hello71) -
Alexandre Julliard -
Erich E. Hoover -
Fabian Maurer