Following the last loadable segment, zero to 15 padding bytes will be added, followed by a checksum byte.
The number of padding bytes is calculated such that the end of the data always falls on a 16-byte boundary (that is, the checksum byte will always be on a 16-byte boundary minus one).
The checksum is a simple XOR of all of the data from each of the segments (the segment headers are not included), also XOR'd with the magic value 0xef.
So if the loader is showing 0xef, this seems to imply the loader is reading all bytes as zero because the flash type is wrong? So has to be related to the configuration byte the loader uses for flash type?
Once I passed the right data into esptool.py's elf2image, it all worked.
ifeq ($(CHIP), 8285)
$(FW_FILE1) $(FW_FILE2) : $(TARGET)
PATH=$(PATH):$(FOLDERPREFIX) $(ESPTOOL_PY) elf2image $(TARGET) --flash_mode dout --flash_freq 80m --flash_size 8m
else
$(FW_FILE1) $(FW_FILE2) : $(TARGET)
PATH=$(PATH):$(FOLDERPREFIX) $(ESPTOOL_PY) elf2image $(TARGET)
endif