The ESP can measure its own supply voltage (you can find lots of examples for that).
Depending on the dropout voltage of your linear regulator, the supply voltage of your ESP will start to decrease when the battery voltage drops below ~3,5V which is still enough to operate for a long time.
Just monitor your ESP supply voltage and consider replacing the batteries when the supply voltage drops below ~3,0V.
This way you won't need any additional circuitry but are still able to detect when the battery goes low early enough (weeks before it has to be replaced).
I've attached a voltage reading of one of my nodes as example.