POC: using SPIFFS to host a large dataset with SRAM indexing
Posted: Sat Feb 11, 2017 7:14 pmI was bored the other week and my mind was wandering: my fingers were traveling the Internet and I was reading tech stuff about OUI: Organizationally Unique Identifier which essentially represent octets 1, 2, and 3 of the MAC.
The article had a link to a web-form where I could input the 3 octets and the "owner" would be revealed; neat. A little searching and I found the .txt file that is the official listing. A little while later, the file was in my downloads folder. A little later, I had it in LibreOffice-calc. A small amount of clean-up and I exported the text file - nearly 800K.
Then, over the next couple of days, I put it in text file into SPIFFS on a NodeMCU and wrote some code to search the dataset for and OUI ... worst-case, 43 seconds!
OK, we know that 43 seconds is not something that any one of us would think is "good", even if the average is 21 - 22 seconds. NOT good.
What I thought would be an hour or so to build an indexing methodology took a lots longer - I had to actually think
The Proof-Of-Concept is complete, the dynamically created index still requires 43 seconds to build the index, but now the OUI owner can be located in a single second, or less!
Code and write-up on my project page.
Direct link.
You DO NOT need to be a member to download the Zip.
Ray
The article had a link to a web-form where I could input the 3 octets and the "owner" would be revealed; neat. A little searching and I found the .txt file that is the official listing. A little while later, the file was in my downloads folder. A little later, I had it in LibreOffice-calc. A small amount of clean-up and I exported the text file - nearly 800K.
Then, over the next couple of days, I put it in text file into SPIFFS on a NodeMCU and wrote some code to search the dataset for and OUI ... worst-case, 43 seconds!
OK, we know that 43 seconds is not something that any one of us would think is "good", even if the average is 21 - 22 seconds. NOT good.
What I thought would be an hour or so to build an indexing methodology took a lots longer - I had to actually think
The Proof-Of-Concept is complete, the dynamically created index still requires 43 seconds to build the index, but now the OUI owner can be located in a single second, or less!
Code and write-up on my project page.
Direct link.
You DO NOT need to be a member to download the Zip.
Ray