Hi,
We have done a few projects now with Revolution Pi and are considering more. One thing I am always asked by management is whether the solid state storage will fail in a short time due to too many writes. In particular we have an upcoming project where there is the potential for frequent writing to the filesystem for status data and log files (e.g. several status writes a minute).
Are we right to be worried about this, or is it a non-issue?
Has anyone encountered problems with this or have advice? Is there data available on this? Do we need to add an external spinning hard drive for this project?
Does the Revolution Pi have wear-levelling built into the storage?
Thanks,
Matthew
Predicting lifetime of eMMC storage
Re: Predicting lifetime of eMMC storage
Yes, the eMMC does use wear leveling per the spec. The following variants are in use:
Flash lifetime is finite. See this presentation for background information. Failure of flash memory e.g. on Tesla cars has been widely reported. However in that particular case, the eMMC was exercised excessively by writing lots of log data and by filling a large portion of the eMMC with static data. We've finetuned logging on the RevPi to reduce the number of writes to /var/log. Also, our image occupies 3 GByte, leaving plenty of space even on the smallest 4 GByte eMMC to apply wear leveling. And there's a button in the RevPi web interface to optionally mount /var/log as an in-memory tmpfs, thus reducing wear of the eMMC at the expense of not having log files available for debugging after a reboot. So far we have never heard of RevPi products failing because of eMMC wear.
I'm afraid it's hard to say if a spinning hard disk makes sense for your application without knowing more details.
- Samsung KLMBG2JETD (Compute Module 3+, 32 GByte, used by RevPi Core 3+ and RevPi Connect +)
- Samsung KLM4G1FEPD (Compute Module 3, 4 GByte, used by Revpi Core 3, replaced by KLM4G1FETE)
- Samsung KLM4G1FEAC (Compute Module 1, 4 GByte, used by RevPi Core)
Flash lifetime is finite. See this presentation for background information. Failure of flash memory e.g. on Tesla cars has been widely reported. However in that particular case, the eMMC was exercised excessively by writing lots of log data and by filling a large portion of the eMMC with static data. We've finetuned logging on the RevPi to reduce the number of writes to /var/log. Also, our image occupies 3 GByte, leaving plenty of space even on the smallest 4 GByte eMMC to apply wear leveling. And there's a button in the RevPi web interface to optionally mount /var/log as an in-memory tmpfs, thus reducing wear of the eMMC at the expense of not having log files available for debugging after a reboot. So far we have never heard of RevPi products failing because of eMMC wear.
I'm afraid it's hard to say if a spinning hard disk makes sense for your application without knowing more details.
Re: Predicting lifetime of eMMC storage
Thanks for that lukas.
My understanding is that the wear levelling means that if we do a lot of small writes (compared to the total space available) then these will be spread out over the available storage. So very roughly speaking, if we overwrote a 1K file 1000000 times, then that would only touch 1GB once at each block.
Is there a way to monitor the wearing of the storage? For example, is the mmc-utils tool mentioned in that presentation available for the RevPi? Or some other tool?
For more context, we are considering implementing network monitoring using nagios on the RevPi. My understanding is that it writes all the status it gathers from the network devices into a file on disk, so that it can be accessed from cgi scripts for the web interface, and it does this (by default) every 15 seconds. And it also writes lots of log files.
Matthew
My understanding is that the wear levelling means that if we do a lot of small writes (compared to the total space available) then these will be spread out over the available storage. So very roughly speaking, if we overwrote a 1K file 1000000 times, then that would only touch 1GB once at each block.
Is there a way to monitor the wearing of the storage? For example, is the mmc-utils tool mentioned in that presentation available for the RevPi? Or some other tool?
For more context, we are considering implementing network monitoring using nagios on the RevPi. My understanding is that it writes all the status it gathers from the network devices into a file on disk, so that it can be accessed from cgi scripts for the web interface, and it does this (by default) every 15 seconds. And it also writes lots of log files.
Matthew
Re: Predicting lifetime of eMMC storage
Yes, mmc-utils can be installed with apt-get and works fine. However "mmc extcsd read /dev/mmcblk0" doesn't output all the attributes listed in the datasheet. In particular, the "Device life time estimation" attributes seem to be missing. I guess the parser would have to be extended.
Honestly I wouldn't worry too much about the Nagios use case. One of our customers is using Icinga (which is derived from Nagios) to monitor hydrogen refueling stations across Europe, see this video for more information. Nicolai, the presenter, has since changed jobs but still occasionally shows up here in the forum.
Honestly I wouldn't worry too much about the Nagios use case. One of our customers is using Icinga (which is derived from Nagios) to monitor hydrogen refueling stations across Europe, see this video for more information. Nicolai, the presenter, has since changed jobs but still occasionally shows up here in the forum.
Re: Predicting lifetime of eMMC storage
Thanks very much for that helpful info lukas. I'll be recommending RevPis for this project.
Re: Predicting lifetime of eMMC storage
My situation is similar the the question posted above. It would be great to have some way to test the status of the eMMC. I also have some questions that I hope someone can fill me in on ...
1) I am curious how the system behaves as the eMMC wears out. Is it like a physical disk where the bad sectors are taken out of use and the disk space slowly reduces in size?
2) My reading on this matter suggests that flash memory is stable once written, and failures appear as bits that do not want to erase, which can be detected by the operating system. Is that correct?
Thanks
1) I am curious how the system behaves as the eMMC wears out. Is it like a physical disk where the bad sectors are taken out of use and the disk space slowly reduces in size?
2) My reading on this matter suggests that flash memory is stable once written, and failures appear as bits that do not want to erase, which can be detected by the operating system. Is that correct?
Thanks
-
p.rosenberger
- Posts: 91
- Joined: 03 Jul 2020, 11:07
Re: Predicting lifetime of eMMC storage
Hi gg94303,
The problem with eMMCs and SD cards is that the manufactures keep almost everything secret. There is (to my knowledge) no reliable way to determine a failing eMMC. How an eMMC handles to many bad cells is up to the controller inside the eMMC. The only way to to handle this problem is to minimize the writes to the eMMC. You can also run a service which checks the data integrity over all files. But this is not so easy implement for most use cases.
Sorry that I can't give you a better answer.
Best Regards
Philipp
The problem with eMMCs and SD cards is that the manufactures keep almost everything secret. There is (to my knowledge) no reliable way to determine a failing eMMC. How an eMMC handles to many bad cells is up to the controller inside the eMMC. The only way to to handle this problem is to minimize the writes to the eMMC. You can also run a service which checks the data integrity over all files. But this is not so easy implement for most use cases.
Sorry that I can't give you a better answer.
Best Regards
Philipp
