Basically you are hitting Wine its virtual memory issues. Try to limit the amount of video memory in the registry to 128MB or so for now. Further there is possibly some Nvidia bug around related to this but I haven't heard back from NV about this yet.