I now have my own X cursor loader, and I have verified that the loaded data is correct. The cursor on the screen is still wrong.
I hacked wlroots to disable support for hardware cursors, and with this version of the library the cursor is displayed correctly.