If your running serial at 115,200 bps, with 512 bytes per universe.
512 bytes per universe = 4,096 bits per universe.
4 universes = 16,384 bits per frame.
10 FPS (100mS) = 160,348 bps.
20 FPS (50mS) = 327,680 bps.
40 FPS (25mS) = 655,360 bps.
Basicaly you will only get 2 full universes in real time over a 115,200 bps serial connection and then only at 10 FPS and thats only including data bits.
SPI is faster, I have been able to read 200 universes per second from an SD card, but I'm guessing your already using SPI for your outpit to the pixels.
The Due has 4 serial ports, it's messy but you could hang 4 ESP's off them and Unicast from Vixen.