pinvokestackimbalance on Windows memset call

[RafaelThome]

We use Blake2b on a WPF application.

We updated the version from 1.1.12 to 2.0.0 and the library started to throw a Managed Debugging Assistant 'PInvokeStackImbalance' on the Isopoh.Cryptography.Blake2B.Create(Blake2BConfig? config, SecureArrayCall secureArrayCall) method.

Though the throw only occurs in debug, it also can cause troubles in release.

I could fix the issue by changing the memset PInvoke Calling Convention to CDecl

That is because ntdll exposes NAPI (Native API), lower level functions than other Win32API DLLs.

Some of the NAPI are also part of the CRT (C Runtime). All of those use the C calling convention (CDECL) instead of STDCALL (the default calling convention for higher level Win32 APIs)

In CDECL, the caller is responsible for freeing stack memory allocated by the call. In STDCALL that responsibility is from the calee.

That leaves the call with an unexpected stack size, which is the reason why the Managed Debugging Assistant throws the 'PInvokeStackImbalance' when debugging.

In a release build the PInvokeStackImbalance is not thrown, but some relate it could cause memory leak or stackoverflow in extreme cases.

Anyway, it is very bad for debugging and it is simply not the correct behavior.

That was tested in a x64 Windows 11, but there is no change in that for other Windows versions.

[apg1034]

I have the same issue if I use Argon2:

Managed Debugging Assistant 'PInvokeStackImbalance'
A call to PInvoke function 'Isopoh.Cryptography.SecureArray!Isopoh.Cryptography.SecureArray.WindowsNative.UnsafeNativeMethods::WindowsMemset' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.