InWorldz
http://inworldz.com/forums/

Why does the cache hate my HDD?
http://inworldz.com/forums/viewtopic.php?f=16&t=23092
Page 1 of 2

Author:  LoganStargazer [ Thu Oct 13, 2016 9:19 am ]
Post subject:  Why does the cache hate my HDD?

This is why I keep my cache on a separate partition. In these pictures a RAMDrive but the same thing happens on a regular HDD.

#1 Empty Cache Partition ready to load Firestorm.
Attachment:
1 - Pre-Viewer Startup.jpg
1 - Pre-Viewer Startup.jpg [ 101.08 KiB | Viewed 2668 times ]


#2 I'm at logged in at home and all textures have been loaded. The drive is already 28% fragmented!
Attachment:
2 - Logged in all textures loaded.jpg
2 - Logged in all textures loaded.jpg [ 170.42 KiB | Viewed 2668 times ]


#3 Fine! I'll optimize it and then TP some place.
Attachment:
3 - Optimized and ready to TP some place.jpg
3 - Optimized and ready to TP some place.jpg [ 176.65 KiB | Viewed 2668 times ]


More pix in next post

Author:  LoganStargazer [ Thu Oct 13, 2016 9:22 am ]
Post subject:  Re: Why does the cache hate my HDD?

#4 I went to Sculpty Republic
Attachment:
4- After one TP.jpg
4- After one TP.jpg [ 173.44 KiB | Viewed 2666 times ]

WTH??? 30% FRAGMENTED!!!!

#5 Optimize again
Attachment:
5- Optimized again.jpg
5- Optimized again.jpg [ 196.05 KiB | Viewed 2666 times ]


#6 Now go back home to where the viewer already knows all the textures.
Attachment:
6 Then back home again.jpg
6 Then back home again.jpg [ 190.8 KiB | Viewed 2666 times ]


20% FRAGMENTED!

This is why I DO NOT keep my cache on the system drive!!!

This is one of the reasons I think that the cache actually slows down the viewer. Unless you are on an SSD or RAMDrive that drive head is going to spend a million years seeking out file fragments.

End of post

Author:  Yichard [ Thu Oct 13, 2016 9:25 am ]
Post subject:  Re: Why does the cache hate my HDD?

NORMALLY fragmentation happens when files are deleted, and the system tries to add new files in the holes. If the new file is bigger than the hole, the system hops to another hole, and the file gets fragmented.

This said, it is astonishing that writing files in an empty disk results in fragmentation. The files should be all appended straigth the one after the other. We need explanations, lol
Maybe there is something wrong with the cache manager. That would explain why it so often has problems.

Getting your second post there: Oh well, there is indeed a problem, lol probably the cache manager plays with the file system itself, and this is why windoze has trouble reading it.
Doing so makes sense, to avoid for instance things like reading copyrighted textures from the cache.

Author:  LoganStargazer [ Thu Oct 13, 2016 10:17 am ]
Post subject:  Re: Why does the cache hate my HDD?

Out of curiosity I reformatted the drive FAT32 (instead of NTFS).
OMG! Look at the FastCache.cache and texture.cache files! Aren't they super important files that need to be accessed a lot on the fly? *cries for my poor drive heads*.

Attachment:
Screenshot_1.jpg
Screenshot_1.jpg [ 185.3 KiB | Viewed 2653 times ]


UPDATE:
I went back and forth between home and Republic a few times. Those two files were close to 1000 fragments each when I checked them again. Can't those two files be kept in RAM and only written to the drive on logout?

Author:  Jim Tarber [ Thu Oct 13, 2016 12:11 pm ]
Post subject:  Re: Why does the cache hate my HDD?

LoganStargazer wrote:
OMG! Look at the FastCache.cache and texture.cache files!
I think you've identified the source of the problem here. The problem is two files growing at small increments in parallel, so they are probably alternating getting the next available new block on the disk.

LoganStargazer wrote:
UPDATE:
I went back and forth between home and Republic a few times. Those two files were close to 1000 fragments each when I checked them again. Can't those two files be kept in RAM and only written to the drive on logout?
That's one technique. Another (which is effectively the same thing) is to buffer changes and only flush it after a significant period of time or data size. The problem with both of these is that a viewer crash might leave the cache in a damaged state, which is something viewers have struggled with. The fragmentation problem you're seeing here may in fact be related to viewer change to avoid cache damage, perhaps by flushing to disk after every little change. This might also explain why the cache is often slower than downloading the asset again from the server over the network. (That should never be, however if they've added a synchronous flush or just forced an unbuffered write after every change, it could kill performance.)

Another technique (Steam installers and other installers use this technique) is to pre-allocate disk space in a single operation so that it reduces fragmented blocks to at last that pre-allocation size. Ideally it could track the largest size of each of those two files seen in the past and then allocate that space in a single new write for future use. That way, in spite of writing small amounts to the file regularly, it would be replacing data in the existing allocated (non-fragmented) block.

But you should really report this to the viewer team (in the Firestorm JIRA since you've done all this analysis with that viewer) by summarizing your findings so that they can investigate and improve it. They may not realize that a "safety" change to avoid cache damage may have killed cache performance (if that's what happened, I'm speculating). Or if it's just been like that all along, it may also be the case for the SL viewer (and you could report it there, or there as well).

Author:  LoganStargazer [ Thu Oct 13, 2016 5:56 pm ]
Post subject:  Re: Why does the cache hate my HDD?

Jim Tarber wrote:
But you should really report this to the viewer team


Why don't you? You can give them tech-talk and point them to this thread maybe.

I have reported cache problems many times and the answer has always been the same: "The cache works perfectly. Firestorm is perfect. The problem is with your computer". I gave up on the FS team a long time ago and don't care even a microscopic bit about LL or anything about Ess Hell in general.

Author:  Jim Tarber [ Thu Oct 13, 2016 6:30 pm ]
Post subject:  Re: Why does the cache hate my HDD?

LoganStargazer wrote:
Jim Tarber wrote:
But you should really report this to the viewer team

Why don't you? You can give them tech-talk and point them to this thread maybe.
Because I have too much on my plate already and I cannot be reporting problems for users with third-party viewers. I don't run Firestorm, and I'm not experiencing this problem as far as I know. It's never good to be someone relaying a problem because there may be followup questions. The person experiencing the trouble should report it. You have adequately documented the problem.
LoganStargazer wrote:
I have reported cache problems many times and the answer has always been the same: "The cache works perfectly. Firestorm is perfect. The problem is with your computer". I gave up on the FS team a long time ago and don't care even a microscopic bit about LL or anything about Ess Hell in general.
I don't really have anything to do with the Firestorm team. If you are unhappy with that viewer and their support, then that's for you to decide. However it seems you now have specific evidence of a problem with the cache design, so they may be more willing to look into it. They can tell you if it is due to a Firestorm-specific change to the general cache code. (I suspect it might be a Firestorm-specific "tweak".) If you've been reporting generalities in the past, these new observations may make a difference.

Author:  LoganStargazer [ Thu Oct 13, 2016 8:50 pm ]
Post subject:  Re: Why does the cache hate my HDD?

Last screenshot. 44% fragmentation after a couple hours. That file has over 2000 fragments!
smh

Attachment:
poop.jpg
poop.jpg [ 238.02 KiB | Viewed 2609 times ]

Author:  Arkady Arkright [ Fri Oct 14, 2016 12:16 am ]
Post subject:  Re: Why does the cache hate my HDD?

Logan, it would be worthwhile seeing if you can reproduce this with the official InWorldz Vn 3 viewer.

Author:  Yichard [ Fri Oct 14, 2016 1:19 am ]
Post subject:  Re: Why does the cache hate my HDD?

Seems that IW3 is obsolete before being complete, with the IW4 waiting for somethin in the SL viewer 4.
But you are still on the point: Inworldz has an occasion to do something.

Anyway I don't believe that a 2000 fragment file is safe. It is anyway very slow, and probably the reason why the viewer frequently jumps several frames.

Page 1 of 2 All times are UTC - 6 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/