Hi,
today I've read an interesting article about ext4: http://www.heise.de/open/Das-Linux-Dateisystem-Ext4--/artikel/138431 (german, but there is also an english version). The new file system seems to be better for large files, as VDR-recordings are. Files up to 512 MB can be saved with just 1 inode, instead of a half Megabyte of inodes in ext3.
So ext4 seems to be perfect for a video-partition, but to make it more perfect, it would be nice if VDR could use the fallocate()-systemcall as mentioned in the article. This would prevent fragmentation in the file system.
For more information you should read the article ;)
Greets, Marcel
Why not use the XFS filesystem? Proven high performance with large files, perfect for an htpc.
On Sun, Jun 7, 2009 at 12:01 AM, Thomas Schäfertschaefer@t-online.de wrote:
Am Sonntag 07 Juni 2009 schrieb VDR User:
Why not use the XFS filesystem? Proven high performance with large files, perfect for an htpc.
He asked for ext4, not for suggestions for filesystems.
If you look closely you'll notice a question mark in what I said. I would hope I don't have to explain any further.
On Sat, 6 Jun 2009 19:11:53 -0700 VDR User user.vdr@gmail.com wrote:
Why not use the XFS filesystem? Proven high performance with large files, perfect for an htpc.
That's what I thought, but I regret using it for my recordings partition now:
* The big performance advantage in deleting large files seems to have vanished, it now seems slower than ext3
* The partition can't be shrunk
* You're less likely to be able to recover data after partition corruption; I once lost an entire partition because its fsck couldn't make it usable again
Marcel Witte wrote:
So ext4 seems to be perfect for a video-partition, but to make it more perfect, it would be nice if VDR could use the fallocate()-systemcall as mentioned in the article. This would prevent fragmentation in the file system.
a) i wouldn't fully trust ext4 for a few more months at least (for a system partition, which you can toss and easily rebuild it's probably ready, for a large, not otherwise backuped, data collection it may not yet be) b) fallocate is relatively new, so you'd require a new libc and kernel (same reason i used fadvise and not sync_file_range) c) libc w/o proper kernel/fs support can emulate fallocate by prewriting zeros to every single block -- not likely what you'd want; this makes above (b) even more relevant.
artur
On 07.06.2009 01:58, Marcel Witte wrote:
So ext4 seems to be perfect for a video-partition, but to make it more perfect, it would be nice if VDR could use the fallocate()-systemcall as mentioned in the article. This would prevent fragmentation in the file system.
Sounds like a good plan, but unfortunately fallocate requires you to know in advance how big a file will be. This is not true for VDR recordings. And if you fallocate with too small or too big sizes, you'll end up with fragmentation or smaller chunks of unused space again. (All in all, this is probably only important for concurrent recordings anyway.)
Cheers,
Udo
On 07.06.2009 01:58, Marcel Witte wrote: So ext4 seems to be perfect for a video-partition, but to make it more perfect, it would be nice if VDR could use the fallocate()-systemcall as mentioned in the article. This would prevent fragmentation in the file
system.
Udo wrote: Sounds like a good plan, but unfortunately fallocate requires you to know
in
advance how big a file will be. This is not true for VDR recordings. And if you fallocate with too small or too big sizes, you'll end up with
fragmentation
or smaller chunks of unused space again. (All in all, this is probably only important for concurrent recordings anyway.)
Well you can predict file size for certain extent. As VDR has the split recording option built in. That is the maximum filesize.
- If you have 1h10min timer. - Allocate 1st file upto split size - Calculate average BW at the same time you are recording - You could even store this - If file is too small, allocate new file for remaining time with average BW + overhead
If you have 10min timer (or short timer which will cause filesize under split size) - if you store average BW what channels are having you could allocate directly estimated size
Naturally this is not 100% accurate, and would cause some big size fragmentation.
For EXT4 it would be nice: - fallocate(4GB) - open file for write - close file after 3GB - automatic fdeallocate(1GB)
jori.hamalainen@teliasonera.com schrieb am Montag 08 Juni 2009:
On 07.06.2009 01:58, Marcel Witte wrote: So ext4 seems to be perfect for a video-partition, but to make it more perfect, it would be nice if VDR could use the fallocate()-systemcall as mentioned in the article. This would prevent fragmentation in the file
system.
Udo wrote: Sounds like a good plan, but unfortunately fallocate requires you to know
in
advance how big a file will be. This is not true for VDR recordings. And if you fallocate with too small or too big sizes, you'll end up with
fragmentation
or smaller chunks of unused space again. (All in all, this is probably only important for concurrent recordings anyway.)
Well you can predict file size for certain extent. As VDR has the split recording option built in. That is the maximum filesize.
- If you have 1h10min timer.
- Allocate 1st file upto split size
- Calculate average BW at the same time you are recording
- You could even store this
- If file is too small, allocate new file for remaining time with average
BW + overhead
If you have 10min timer (or short timer which will cause filesize under split size)
- if you store average BW what channels are having you could allocate
directly estimated size
Naturally this is not 100% accurate, and would cause some big size fragmentation.
For EXT4 it would be nice:
- fallocate(4GB)
- open file for write
- close file after 3GB
- automatic fdeallocate(1GB)
This was exactly the idea I had... You know the average bitrate and the timer- length, or if used the length of one "split-file". And I think 99% of all recordings will not be aborted while recording. So this would be the best way to make use of the ext4-extends.
And because of the new libc/kernel you need: You can use #defines ;) also we have a development branch (1.7.x) an until this goes stable I think ext4 is a standard-file system (Fedora, Ubuntu and openSUSE will use it as default in the next versions)
On 06/09/09 00:47, Marcel Witte wrote:
jori.hamalainen@teliasonera.com schrieb am Montag 08 Juni 2009:
On 07.06.2009 01:58, Marcel Witte wrote: So ext4 seems to be perfect for a video-partition, but to make it more perfect, it would be nice if VDR could use the fallocate()-systemcall as mentioned in the article. This would prevent fragmentation in the file
system.
Udo wrote: Sounds like a good plan, but unfortunately fallocate requires you to know
in
advance how big a file will be. This is not true for VDR recordings. And if you fallocate with too small or too big sizes, you'll end up with
fragmentation
or smaller chunks of unused space again. (All in all, this is probably only important for concurrent recordings anyway.)
Well you can predict file size for certain extent. As VDR has the split recording option built in. That is the maximum filesize.
- If you have 1h10min timer.
- Allocate 1st file upto split size
- Calculate average BW at the same time you are recording
- You could even store this
- If file is too small, allocate new file for remaining time with average
BW + overhead
If you have 10min timer (or short timer which will cause filesize under split size)
- if you store average BW what channels are having you could allocate
directly estimated size
Naturally this is not 100% accurate, and would cause some big size fragmentation.
For EXT4 it would be nice:
- fallocate(4GB)
- open file for write
- close file after 3GB
- automatic fdeallocate(1GB)
This was exactly the idea I had... You know the average bitrate and the timer- length, or if used the length of one "split-file". And I think 99% of all recordings will not be aborted while recording. So this would be the best way to make use of the ext4-extends.
And because of the new libc/kernel you need: You can use #defines ;) also we have a development branch (1.7.x) an until this goes stable I think ext4 is a standard-file system (Fedora, Ubuntu and openSUSE will use it as default in the next versions)
Just my 2ct: I find it strange that an application would have to "preallocate" disk space etc. The file system should be able to handle these things in an efficient way. It should not matter to VDR which file system a particular machine actually uses. If one file system is better at handling large files, that's fine, but don't force VDR to "know" about this...
Klaus
On 07.06.2009 01:58, Marcel Witte wrote:
So ext4 seems to be perfect for a video-partition, but to make it more perfect, it would be nice if VDR could use the fallocate()-systemcall as mentioned in the article. This would prevent fragmentation in the file system.
Sounds like a good plan, but unfortunately fallocate requires you to know in advance how big a file will be. This is not true for VDR recordings. And if you fallocate with too small or too big sizes, you'll end up with fragmentation or smaller chunks of unused space again. (All in all, this is probably only important for concurrent recordings anyway.)
Cheers,
Udo
-
Artur Skawina -
jori.hamalainen＠teliasonera.com -
Klaus Schmidinger -
Marcel Witte -
Thomas Schäfer -
Tony Houghton -
Udo Richter -
VDR User