[CODE SNIPPET] How I do volatile and persistent storage
Posted: Sat Apr 06, 2019 7:04 pmThis seems to be something that people need to do a lot, and in response to @eriksl posting up that he needed to change his own routines, I've decided to release my module.
This code implements volatile and persistent storage for the ESP8266. The persistent storage reads and writes key/value pairs and uses a journal file system kind of approach - when you write data with the same key, the old key is wiped (using the trick that in NAND flash you can successfully overwrite a memory location with zeros but not ones), and the newer value is written to the end of storage. The code maintains a round robin of flash sectors, and when a flash sector is full, the code moves on to the next one. When it moves onto the last available free flash sector, it goes back to the oldest used sector, moves any active records to the new one, and then erases it so that there's always at least one free sector waiting.
The volatile storage is much simpler and just reads and writes the entire user area in RTC memory at once.
The calling code is responsible for any caching of data read from either system. For volatile memory I just have a structure that looks like this:
and I add the various subsystem's volatile info into it as needed, and then just use pointers to that to play with the data.
For persistent info I set up a set of pointers to the data and retrieve as needed:
So my code to get the data looks like this - note that in the persistent storage section, I am also setting up system defaults if the data doesn't exist in the store, and then I only write the record if it gets changed from the system default:
I've also included code for doing 32bit CRC checks - that's the only other dependency. Replace if you have your own already - mine is as fast as you can get without using static table lookups (I built it for size and OK speed)
As always comments and improvement suggestions are always appreciated!
This code implements volatile and persistent storage for the ESP8266. The persistent storage reads and writes key/value pairs and uses a journal file system kind of approach - when you write data with the same key, the old key is wiped (using the trick that in NAND flash you can successfully overwrite a memory location with zeros but not ones), and the newer value is written to the end of storage. The code maintains a round robin of flash sectors, and when a flash sector is full, the code moves on to the next one. When it moves onto the last available free flash sector, it goes back to the oldest used sector, moves any active records to the new one, and then erases it so that there's always at least one free sector waiting.
The volatile storage is much simpler and just reads and writes the entire user area in RTC memory at once.
The calling code is responsible for any caching of data read from either system. For volatile memory I just have a structure that looks like this:
Code: Select all
typedef struct VolatileInfo {
/* ... */
WiFiInfoV_t wifi;
TimerInfoV_t timers;
} VolatileInfo_t;
static VolatileInfo_t volatile_info;
static bool volatile_retrieved = false;and I add the various subsystem's volatile info into it as needed, and then just use pointers to that to play with the data.
For persistent info I set up a set of pointers to the data and retrieve as needed:
Code: Select all
typedef struct PersistentInfo {
/* ... */
WiFiInfoP_t *wifi;
TimerInfoP_t *timers;
/* ... */
} PersistentInfo_t;
static PersistentInfo_t persistent_info = {NULL, NULL, /* ... */};
So my code to get the data looks like this - note that in the persistent storage section, I am also setting up system defaults if the data doesn't exist in the store, and then I only write the record if it gets changed from the system default:
Code: Select all
void * ICACHE_FLASH_ATTR sysinfo_get(uint32_t id) {
if (STORE(id) == STORAGE_VOLATILE) {
if (volatile_retrieved == false) {
uint8_t data[512];
int32_t len = store_volatile_read(data);
INFO("SYSINFO", "read volatile store, len = %d", len);
if (len > 0) {
os_memcpy(&volatile_info, data, len);
} else {
os_memset(&volatile_info, 0, sizeof(VolatileInfo_t));
}
volatile_retrieved = true;
}
switch (SUBSYSTEM(id)) {
case SS_WIFI:
return &volatile_info.wifi;
break;
/* ... */
}
}
switch (SUBSYSTEM(id)) {
case SS_WIFI:
if (persistent_info.wifi == NULL) {
persistent_info.wifi = os_zalloc(sizeof(WiFiInfoP_t));
int32_t size = store_persist_exists(id);
if (size == 0 || store_persist_read(id, persistent_info.wifi) != size) {
os_memcpy(persistent_info.wifi, &static_wifi_list, sizeof(WiFiInfoP_t));
}
}
return persistent_info.wifi;
break;
/* ... */
}
}
void ICACHE_FLASH_ATTR sysinfo_write_if_dirty(uint32_t id) {
if (STORE(id) == STORAGE_VOLATILE) {
INFO("SYSINFO", "write volatile store, len = %d", store_volatile_write(&volatile_info, sizeof(VolatileInfo_t)));
} else {
uint32_t len = 0;
switch (SUBSYSTEM(id)) {
case SS_WIFI:
len = sizeof(WiFiInfoP_t);
if (os_memcmp(&persistent_info.wifi, &static_wifi_list, len) != 0) {
INFO("SYSINFO", "write wifi info, len = %d", store_persist_write(id, persistent_info.wifi, len));
}
break;
/* ... */
}
}
}
I've also included code for doing 32bit CRC checks - that's the only other dependency. Replace if you have your own already - mine is as fast as you can get without using static table lookups (I built it for size and OK speed)
As always comments and improvement suggestions are always appreciated!