InWorldz

Where your Dreams are our Vision!

It is currently Sat Jul 21, 2018 11:30 am | All times are UTC - 6 hours
 Page 1 of 2 [ 20 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Why does the cache hate my HDD?
PostPosted: Thu Oct 13, 2016 9:19 am 
User avatar

Joined: Mon Mar 09, 2015 7:14 am
Posts: 938
This is why I keep my cache on a separate partition. In these pictures a RAMDrive but the same thing happens on a regular HDD.

#1 Empty Cache Partition ready to load Firestorm.
Attachment:
1 - Pre-Viewer Startup.jpg
1 - Pre-Viewer Startup.jpg [ 101.08 KiB | Viewed 2667 times ]


#2 I'm at logged in at home and all textures have been loaded. The drive is already 28% fragmented!
Attachment:
2 - Logged in all textures loaded.jpg
2 - Logged in all textures loaded.jpg [ 170.42 KiB | Viewed 2667 times ]


#3 Fine! I'll optimize it and then TP some place.
Attachment:
3 - Optimized and ready to TP some place.jpg
3 - Optimized and ready to TP some place.jpg [ 176.65 KiB | Viewed 2667 times ]


More pix in next post


Last edited by LoganStargazer on Thu Oct 13, 2016 9:23 am, edited 1 time in total.


_________________
Logan's Stuff And Junk http://tp2.me/LOGANSSTUFFANDJUNK
My Youtube Channel - https://www.youtube.com/channel/UC2I0Zb-io_DK1argPuo4RyA
Jesus loves you. Everybody else thinks you suck.
Offline
 Profile  
 
 Post subject: Re: Why does the cache hate my HDD?
PostPosted: Thu Oct 13, 2016 9:22 am 
User avatar

Joined: Mon Mar 09, 2015 7:14 am
Posts: 938
#4 I went to Sculpty Republic
Attachment:
4- After one TP.jpg
4- After one TP.jpg [ 173.44 KiB | Viewed 2665 times ]

WTH??? 30% FRAGMENTED!!!!

#5 Optimize again
Attachment:
5- Optimized again.jpg
5- Optimized again.jpg [ 196.05 KiB | Viewed 2665 times ]


#6 Now go back home to where the viewer already knows all the textures.
Attachment:
6 Then back home again.jpg
6 Then back home again.jpg [ 190.8 KiB | Viewed 2665 times ]


20% FRAGMENTED!

This is why I DO NOT keep my cache on the system drive!!!

This is one of the reasons I think that the cache actually slows down the viewer. Unless you are on an SSD or RAMDrive that drive head is going to spend a million years seeking out file fragments.

End of post


Last edited by LoganStargazer on Thu Oct 13, 2016 9:26 am, edited 1 time in total.


_________________
Logan's Stuff And Junk http://tp2.me/LOGANSSTUFFANDJUNK
My Youtube Channel - https://www.youtube.com/channel/UC2I0Zb-io_DK1argPuo4RyA
Jesus loves you. Everybody else thinks you suck.
Offline
 Profile  
 
 Post subject: Re: Why does the cache hate my HDD?
PostPosted: Thu Oct 13, 2016 9:25 am 
User avatar

Joined: Fri Jul 29, 2011 2:10 pm
Posts: 3989
Location: France
NORMALLY fragmentation happens when files are deleted, and the system tries to add new files in the holes. If the new file is bigger than the hole, the system hops to another hole, and the file gets fragmented.

This said, it is astonishing that writing files in an empty disk results in fragmentation. The files should be all appended straigth the one after the other. We need explanations, lol
Maybe there is something wrong with the cache manager. That would explain why it so often has problems.

Getting your second post there: Oh well, there is indeed a problem, lol probably the cache manager plays with the file system itself, and this is why windoze has trouble reading it.
Doing so makes sense, to avoid for instance things like reading copyrighted textures from the cache.



_________________
Elfs are in virtual worlds to share a vision of a wonderful life :-)
Elven bards and storytellers
"Elf Dream" for Elf living
"Shedrupling University" science
YichTrains Elven Works Lysaer Gortax Robots
Offline
 Profile  
 
 Post subject: Re: Why does the cache hate my HDD?
PostPosted: Thu Oct 13, 2016 10:17 am 
User avatar

Joined: Mon Mar 09, 2015 7:14 am
Posts: 938
Out of curiosity I reformatted the drive FAT32 (instead of NTFS).
OMG! Look at the FastCache.cache and texture.cache files! Aren't they super important files that need to be accessed a lot on the fly? *cries for my poor drive heads*.

Attachment:
Screenshot_1.jpg
Screenshot_1.jpg [ 185.3 KiB | Viewed 2652 times ]


UPDATE:
I went back and forth between home and Republic a few times. Those two files were close to 1000 fragments each when I checked them again. Can't those two files be kept in RAM and only written to the drive on logout?



_________________
Logan's Stuff And Junk http://tp2.me/LOGANSSTUFFANDJUNK
My Youtube Channel - https://www.youtube.com/channel/UC2I0Zb-io_DK1argPuo4RyA
Jesus loves you. Everybody else thinks you suck.
Offline
 Profile  
 
 Post subject: Re: Why does the cache hate my HDD?
PostPosted: Thu Oct 13, 2016 12:11 pm 
User avatar

Joined: Mon Jun 07, 2010 7:07 pm
Posts: 7936
Location: Nova Scotia, Canada
LoganStargazer wrote:
OMG! Look at the FastCache.cache and texture.cache files!
I think you've identified the source of the problem here. The problem is two files growing at small increments in parallel, so they are probably alternating getting the next available new block on the disk.

LoganStargazer wrote:
UPDATE:
I went back and forth between home and Republic a few times. Those two files were close to 1000 fragments each when I checked them again. Can't those two files be kept in RAM and only written to the drive on logout?
That's one technique. Another (which is effectively the same thing) is to buffer changes and only flush it after a significant period of time or data size. The problem with both of these is that a viewer crash might leave the cache in a damaged state, which is something viewers have struggled with. The fragmentation problem you're seeing here may in fact be related to viewer change to avoid cache damage, perhaps by flushing to disk after every little change. This might also explain why the cache is often slower than downloading the asset again from the server over the network. (That should never be, however if they've added a synchronous flush or just forced an unbuffered write after every change, it could kill performance.)

Another technique (Steam installers and other installers use this technique) is to pre-allocate disk space in a single operation so that it reduces fragmented blocks to at last that pre-allocation size. Ideally it could track the largest size of each of those two files seen in the past and then allocate that space in a single new write for future use. That way, in spite of writing small amounts to the file regularly, it would be replacing data in the existing allocated (non-fragmented) block.

But you should really report this to the viewer team (in the Firestorm JIRA since you've done all this analysis with that viewer) by summarizing your findings so that they can investigate and improve it. They may not realize that a "safety" change to avoid cache damage may have killed cache performance (if that's what happened, I'm speculating). Or if it's just been like that all along, it may also be the case for the SL viewer (and you could report it there, or there as well).


Offline
 Profile  
 
 Post subject: Re: Why does the cache hate my HDD?
PostPosted: Thu Oct 13, 2016 5:56 pm 
User avatar

Joined: Mon Mar 09, 2015 7:14 am
Posts: 938
Jim Tarber wrote:
But you should really report this to the viewer team


Why don't you? You can give them tech-talk and point them to this thread maybe.

I have reported cache problems many times and the answer has always been the same: "The cache works perfectly. Firestorm is perfect. The problem is with your computer". I gave up on the FS team a long time ago and don't care even a microscopic bit about LL or anything about Ess Hell in general.



_________________
Logan's Stuff And Junk http://tp2.me/LOGANSSTUFFANDJUNK
My Youtube Channel - https://www.youtube.com/channel/UC2I0Zb-io_DK1argPuo4RyA
Jesus loves you. Everybody else thinks you suck.
Offline
 Profile  
 
 Post subject: Re: Why does the cache hate my HDD?
PostPosted: Thu Oct 13, 2016 6:30 pm 
User avatar

Joined: Mon Jun 07, 2010 7:07 pm
Posts: 7936
Location: Nova Scotia, Canada
LoganStargazer wrote:
Jim Tarber wrote:
But you should really report this to the viewer team

Why don't you? You can give them tech-talk and point them to this thread maybe.
Because I have too much on my plate already and I cannot be reporting problems for users with third-party viewers. I don't run Firestorm, and I'm not experiencing this problem as far as I know. It's never good to be someone relaying a problem because there may be followup questions. The person experiencing the trouble should report it. You have adequately documented the problem.
LoganStargazer wrote:
I have reported cache problems many times and the answer has always been the same: "The cache works perfectly. Firestorm is perfect. The problem is with your computer". I gave up on the FS team a long time ago and don't care even a microscopic bit about LL or anything about Ess Hell in general.
I don't really have anything to do with the Firestorm team. If you are unhappy with that viewer and their support, then that's for you to decide. However it seems you now have specific evidence of a problem with the cache design, so they may be more willing to look into it. They can tell you if it is due to a Firestorm-specific change to the general cache code. (I suspect it might be a Firestorm-specific "tweak".) If you've been reporting generalities in the past, these new observations may make a difference.


Offline
 Profile  
 
 Post subject: Re: Why does the cache hate my HDD?
PostPosted: Thu Oct 13, 2016 8:50 pm 
User avatar

Joined: Mon Mar 09, 2015 7:14 am
Posts: 938
Last screenshot. 44% fragmentation after a couple hours. That file has over 2000 fragments!
smh

Attachment:
poop.jpg
poop.jpg [ 238.02 KiB | Viewed 2608 times ]



_________________
Logan's Stuff And Junk http://tp2.me/LOGANSSTUFFANDJUNK
My Youtube Channel - https://www.youtube.com/channel/UC2I0Zb-io_DK1argPuo4RyA
Jesus loves you. Everybody else thinks you suck.
Offline
 Profile  
 
 Post subject: Re: Why does the cache hate my HDD?
PostPosted: Fri Oct 14, 2016 12:16 am 
User avatar

Joined: Mon Jun 14, 2010 3:08 am
Posts: 5759
Location: Wales
Logan, it would be worthwhile seeing if you can reproduce this with the official InWorldz Vn 3 viewer.



_________________
Ark
Good judgement comes from experience, experience comes from bad judgement
Offline
 Profile  
 
 Post subject: Re: Why does the cache hate my HDD?
PostPosted: Fri Oct 14, 2016 1:19 am 
User avatar

Joined: Fri Jul 29, 2011 2:10 pm
Posts: 3989
Location: France
Seems that IW3 is obsolete before being complete, with the IW4 waiting for somethin in the SL viewer 4.
But you are still on the point: Inworldz has an occasion to do something.

Anyway I don't believe that a 2000 fragment file is safe. It is anyway very slow, and probably the reason why the viewer frequently jumps several frames.



_________________
Elfs are in virtual worlds to share a vision of a wonderful life :-)
Elven bards and storytellers
"Elf Dream" for Elf living
"Shedrupling University" science
YichTrains Elven Works Lysaer Gortax Robots
Offline
 Profile  
 
Display posts from previous:  Sort by  
 Page 1 of 2 [ 20 posts ]  Go to page 1, 2  Next


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Site Navigation