Answering my own question.
When you return an IStream from your own COM server the .NET interop DLL puts an IStream into it s interface. For example lets say the type library is MyComServer, then the interop will contain a class MyComServer.Interop.IStream.
this IStream class has functions like RemoteRead, RemoteWrite etc. These take a ref byte as the first parameter. Using these worked fine on 32bit to 32bit, but that ref probably becomes a pointer and so on 64 to 32 something went wrong.
The solution is to convert the MyComServer.Interop.IStream to System.Runtime.InteropServices.ComTypes.IStream using "as" (or just cast probably). This then takes a more familiar byte[], int count and IntPtr for the return size.
The IntPtr is annoying as it could be an out int, but I used (typing this in by hand so not compiler checked..)
byte[] buffer = new byte[100];
IntPtr ptr = Marshall.AllocHGlobal( sizeof(int) );
stream.Read( buffer, 100, ptr );