AFAIK the ESP has the modes STA (station), AP (access point) and STA + AP (dual mode).
Since the ESP only has hardware for only one channel/mode, I can imagine that STA + AP uses software for the AP-mode (hence the name "SoftAP"), whilst in the other modes only hardware is used (as either STA or AP).
If my assumption is correct, STA- or AP-mode alone would be fastest/most efficient.
So if you need both modes at the same time, you'll have to accept a little bit speed/efficiency decrease.
But... I'm only guessing here.