Random crashes when using Clang-CL to compile 32-bit binaries

in #development2 years ago

As I was reviewing commits on one project, I noticed that after they added Clang-CL builds to continuous integration (CI), the 32-bit build crashed with access violation at address 0xFFFFFFFF. There was no code that would result in NULL (0x00000000) access, or access to one byte before any allocated memory.

After looking up on internet, address 0xFFFFFFFF equals to error code "Invalid handle", essentially value that should never be valid. That gave me a hint that something invalid was passed in the generated code... The offending instruction was movaps, which means "move aligned packed single-precision floating point numbers". When looking at the address where it tried to save the register contents, the last hexadecimal number of the address was 8, when it obviously should have been 0 for a vector instruction.

That meant that the code was storing the struct in misaligned memory location and when the offset to struct member was multiple of 16, it tried to modify more than one struct member at once as all the values were constants, combining the raw bytes using a vector register.

The solution that worked was to align the start of the struct at 16-byte boundary instead of the default 8-byte boundary.