WHY????? Why does SPIFFS slow down....

Your new topic does not fit any of the above??? Check first. Then post here. Thanks.

Moderator: igrr

 Reply

User avatar

WHY????? Why does SPIFFS slow down....

By RichardS
- Thu Mar 08, 2018 7:37 pm #74482 the more you write files, the slower it gets!!!!! Its insane, I have been battling this from 2.0.0 to 2.4.0 its always the same, I have changed code around, and code logic, etc....

Can not make it work.

Run this Arduino code (it will format your SPIFFS) and you will see it get slower and slower......

WHY??

Code: Select all//******************************************************************************
// SPIFFS TEST
//******************************************************************************

#include <FS.h>

void setup() {
  Serial.begin(115200);
  SPIFFS.begin();
}

void removeAll() {
#if 0
  SPIFFS.remove("/1");
  SPIFFS.remove("/2");
  SPIFFS.remove("/3");
  SPIFFS.remove("/4");
  SPIFFS.remove("/5");
#endif 
  SPIFFS.remove("/5");
  SPIFFS.remove("/4");
  SPIFFS.remove("/3");
  SPIFFS.remove("/2");
  SPIFFS.remove("/1");
}

#define SECTOR 256
char buf[SECTOR];
void make(String f, int n) {
  int time;
  int packet = 0;
 
  Serial.printf("making %s length = %d\n", f.c_str(), n);

//  SPIFFS.remove(f);

  File pESP = SPIFFS.open(f, "w");
  time = millis();
  while (n > 0) {
    if((packet++ % 20) == 0) {
      Serial.println(millis()-time);
      time = millis();
    }
    int bytes = (n > SECTOR) ? SECTOR : n;
    n -= SECTOR;
    //memset(buf, n, sizeof(buf));
    pESP.write((uint8_t*)buf, bytes);
    yield();
  }
  pESP.close();
}

void loop() {
  uint32_t first, second, third, fourth;

  Serial.printf("\nFormatting...\n");
  SPIFFS.format();

  if(0) {
    // was used to read and write at same time for tests...
   
    //make("/temp", 1234 + 12112 + 200220 + 454030 + 20220 + 100100);
 
    //fp = SPIFFS.open("temp", "r");
    //fp.seek(0, SeekEnd);
    //fp.position();
    //fp.seek(0, SeekSet);
  }

  Serial.println("first run");
 
  uint32_t startTime = millis();
  make("/1", 1234);
  make("/2", 12112);
  make("/3", 200220);
  make("/4", 454030);
  make("/5", 20220);
  make("/6", 100100);
  first = millis() - startTime;

  removeAll();
  Serial.println("second run");
 
  startTime = millis();
  make("/1", 1234);
  make("/2", 12112);
  make("/3", 200220);
  make("/4", 454030);
  make("/5", 20220);
  make("/6", 100100);
  second = millis() - startTime;

  removeAll();
  Serial.println("third run");
 
  startTime = millis();
  make("/1", 1234);
  make("/2", 12112);
  make("/3", 200220);
  make("/4", 454030);
  make("/5", 20220);
  make("/6", 100100);
  third = millis() - startTime;

  removeAll();
  Serial.println("fourth run");
 
  startTime = millis();
  make("/1", 1234);
  make("/2", 12112);
  make("/3", 200220);
  make("/4", 454030);
  make("/5", 20220);
  make("/6", 100100);
  fourth = millis() - startTime;

  Serial.printf("Time first = %dms\n", first);
  Serial.printf("Time second = %dms\n", second);
  Serial.printf("Time third = %dms\n", third);
  Serial.printf("Time fourth = %dms\n", fourth);
  for(;;) yield();
}
You do not have the required permissions to view the files attached to this post.
User avatar

Re: WHY????? Why does SPIFFS slow down....

By RichardS
- Fri Mar 09, 2018 12:51 pm #74494 Turns out its not a bug for sure.

With lots of investigative work into the inner code of the SPIFFS I can see that it at some time will attempt to do "garbage collection" on the flash.... I was able to find a file called spiffs_config.h and it had a bunch of defines for debug message, I found one for garbage collection and turned it on....

What I see now is the moment things start slowing down, it starts to call the garbage collection routines. So all this time battling the slow down I was seeing was not my code... it is the nature of SPIFFS....

That said I am still a little completed as it seems to be doing a lot of collection on what should be not that much garbage, at least in my sample code and tests I have conducted.

Anyone who wants to help solve slow garbage collection is welcome to join in!

RichardS
User avatar

Re: WHY????? Why does SPIFFS slow down....

By rudy
- Fri Mar 09, 2018 6:57 pm #74502 I saw your post last night as I was about to go to bed. I decided to try it, but I didn't really take a good look at the file sizes. (maybe I did but I can't remember it now) I would think that garbage collection should only be happening if you have a large portion of the space used.

make("/1", 1234);
make("/2", 12112);
make("/3", 200220);
make("/4", 454030);
make("/5", 20220);
make("/6", 100100);

OK, looking at it now, if these are the file sizes then I do think garbage collection may be validly called.
User avatar

Re: WHY????? Why does SPIFFS slow down....

By RichardS
- Sun Mar 11, 2018 8:51 am #74524 Yes it maybe validly called, however, its very very slow, I upload a one file that basically contains mainly other files, then I unpack them and write them, its for an upgrade process, and the garbage collection effects the speed greatly!

What should take 30 seconds or less can take near 10 minutes!

I am hoping we can either make a faster garbage collection routine, or be able to delay the collection, and when you know the ESP has spare time to deal with it, then the process can be done then.....

RichardS
 Reply
Return to General Discussions