It's probably back EMF from the relay triggering the reset pin, your relay is probably not isolated well enough in terms of EM interference.
Most of the projects out there I've seen have used an opto isolator for relays, that way you have no electrical connection between the relay and the ESP8266, no signal path, no interference and therefore, no spontaneous resets.
Do you have a diode in reverse polarity across the relay coil? This will help short any back-EMF when the relay is toggled.
What does your power supply to the module and the relay look like? A surge in current draw can reset the ESP8266 if the supply can't keep up with the dual instantaneous demands of energizing a relay and running the module.
How well isolated is the ESP8266's ground supply from the relay's ground? Once again, it's a potential signal path for spikes and other power weirdness due to driving an inductive load such as a relay.
These are all questions you should be asking when it comes to Microcontrollers and relays in general.