P/Invoke … evil

Okay, so I didn’t end up getting back to my list of .Net 4 beta 2 updates, but I thought I would write something about the P/Invoke hell I went through today.

Somewhere I managed to get an incorrect definition of CERT_EXTENSION with 4 fields, string (Marshal as LPStr), int, int and IntPtr.
I subsequently saw that CERT_EXTENSION was actually 3 fields, string, int and struct { int, IntPtr }. However it did not occur to me that this was a problem. Everything was set to layout sequential, so I thought it was all fine.

Obviously given I am writing this it was not all fine. 4 hours later, having written a small ASN.1 encoder, and fiddling with different ASN.1 formats, trying 19 different ways to marshal everything and writing a c++ program just to verify the API itself was not broken, I changed it and presto, it works. I guess field alignment for struct is greater than int…

Ahh, found it. Because the Struct contains an IntPtr which is 64bit the alignment of the struct itself is 64bit, rather than the 32bit of the int. So with the struct definition and a 64bit environment there are two extra 32bit runs of padding in the correct version as opposed to the incorrect version. (I think…)