Doing it with a small server acting as the internet facing side will make your smart home more secure. You don't want people to turn your lights on and off.
You could ad password protection to your devices but it would not be too fun to have to talk to each device on a different port number.
http://esp8266basic.com
A BASIC interpreter for your ESP
Working now on Autodrop3d. A 3d printer with automatic part ejection system. https://autodrop3d.com