As I understand things, "github" is an active code repository where developers can 'fork' the code, work on it independently to enhance or correct specific bug(s), and then issuing a 'pull request' back to igrr to have that code incorporated into the staging version of the code and if no issues are reported, eventually into the master version of the code.
For most ArduinoIDE users, they should utilize Board Manger to install the core as documented. For Arduino developers, they may wish to run with the "staging" version which is more current, but not fully tested. Eventually, the staging version is "promoted" into the master and the process starts again.
The XML files, such as
http://arduino.esp8266.com/staging/package_esp8266com_index.json identify to the ArduinoIDE how to install both the core and the specific tools required to permit the ArduinoIDE click to compile and upload feature. A knowledgeable Arduinonite could download the ZIP and manually install the core, then download the tools and manually install them... but the process is error-prone for the common Arduino user.
Ray