I have a feeling that one of the issues at higher speed is making sure all of the relevant pins are pulled high or low, because otherwise you seem to get a lot of failures. On the board I've made it now reliably flashes every time at 921600.
I'll have to try out timing it at the different rates to see what difference it makes though.