I would suggest that you a) always have a battery connected and b) use a low-drop-out regulator to provide your 3.3V to the ESP.
My problem is slightly different in that I need the ESP to be on in a standby/monitoring mode on battery power only for many hours. The problem is, if I try to use SLEEP, it will reboot on start-up - and take 30 sec - 1 minute to do so! What I really want is to be able to put it to sleep for 1 second, wake it up to check stuff for about 5mS then go back to sleep again! And all whilst using 5mA or less for 95% of the time!
Phil