 |
 |
 |
|
 |
You are not logged in.
Pages: 1 2
Hi,
I've had some major troubles with my little EEE's filesystem in the past few days, have searched the forums quite a bit but haven't found an answer to the problem yet.
I think my long-term strategy is to remove the unionfs and go with straight ext2, but it's the last week of Uni this week so if possible would love to find a "quick fix" until I have time to do a proper upgrade in the holidays!
The sympoms are these: a few days ago, I booted my EEE-PC and things like Wireless failed to start and gave error messages. Investigating in the terminal I find the cause seems to be "no space left on device", e.g. trying "/etc/init.d/networking restart" eventually fails with a message "cannot touch '/etc/hosts.new': No space left on device".
THis is happening despite the output of df -h which is:
Filesystem Size Used Avail Use% Mounted on
rootfs 1.4G 727M 623M 54% /
/dev/sda1 1.4G 727M 623M 54% /
unionfs 1.4G 727M 623M 54% /
tmpfs 249M 4.0K 249M 1% /dev/shm
tmpfs 128M 12K 128M 1% /tmp
/dev/sdhc1 7.5G 707M 6.5G 10% /home/user/homeSDHC
/dev/sdc1 3.9G 215M 3.6G 6% /media/Cruzer Crossfire
and typing mount gives the following which looks pretty normal:
rootfs on / type rootfs (rw)
/dev/sda1 on / type ext2 (ro)
unionfs on / type unionfs (rw,dirs=/=rw:/=ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw)
tmpfs on /dev/shm type tmpfs (rw)
tmpfs on /tmp type tmpfs (rw)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sdhc1 on /home/user/homeSDHC type ext2 (rw)
/dev/sdc1 on /media/Cruzer Crossfire type vfat (rw,fmask=0022,dmask=0022,codepage=cp850,iocharset=iso8859-1)
I recently tried booting up using a "TRK rescue" Linux boot USB disk I created a while back, and deleting some non-essential files off the "protected" HDD (eg /usr/share/games), but it didn't seem to help.
Any ideas/help greatly appreciated! 
Offline
Hi
I've just had a similar problem. For me it turned out to be the iNodes.... do 'df -i' and see if you have zero or low number of IFree. I was on between zero and 20. I found out lots of zero size .wh.? files are created when you delete things for UnionFS I think. I ended up doing what you did and booting off of Parted Magic and deleted some .wh files as it was too unstable to use file manager (9000 in my opera cache directory).... still in the process of deleting now but running a lot smoother.
Hope this helps ... oh i did a 'ls -alR |more' to give me some idea where there was lots of similar files.
Last edited by M1ster_Happy (2008-06-03 1:50:42 am)
Offline
Thanks M1ster_Happy, you are a champ! Yes, Inodes was down to zero, and as you suggested lots of .wh. files was a big culprit- for me it was mainly in the /var/tmp directory, lots of sqlite files it seems.
Surprising that more people haven't run into this problem?? Maybe we should create a Wiki entry.
Oh well, it's great to be up & running again, & reinforces my plan to get rid of unionfs when I have more tinkering time.
Offline
Yeah I just found the tmp ones after the post .... there was ... alot! I was hoping the files would be visible when booting normally but it seems I have to boot off of Parted Magic or another live version to see/delete them.
To be honest I wouldn't want to recommend it in a wiki as I'm a total linux newbie so it may turn out to be wrong! (I hope not) For now if you can change the thread name you could prefix SOLVED on it, just an idea.
I'm not really sure why they get created as they seem to be a 'white out' file to hide the deleted file on the lower partition of the unionfs set up... but they get created in directories that don't exist on the lower partition, and in directories that have '.wh.__dir_opaque' files which is suggesting lower partition directories to be ignored anyway (so i've read.. maybe wrong! ). I'm sure there's some logic to it but the way it seems it should be fine to delete any '.wh.*' files where a '.wh.__dir_opaque' file exists (not deleting the opaque), and where the lower partitions directory is empty or does not exist. I think you have to watch out for deleting a deleted directory file as well which is 'hiding' the directory in existence on the lower partition (don't think its too much of a problem as it will just be visible again).
I'd like to write a script to do it but like you I don't believe I need the unionfs system so I think I will just remove it.... but until then it seems deleting them will keep us going. Like you said you would think more people would have the same issue... but then alot of mine were opera so i guess I wouldn't have hit the issue for a while longer... guess we'll see.
Let me know if you get any issues.
Offline
Hmmm I have found similar stuff overhere:
/home/user> df -h Filesystem Size Used Avail Use% Mounted on rootfs 1.4G 452M 898M 34% / /dev/sda1 1.4G 452M 898M 34% / unionfs 1.4G 452M 898M 34% / tmpfs 443M 16K 443M 1% /dev/shm tmpfs 128M 56K 128M 1% /tmp
While diskutil shows use 451MB (31.76%) and available 969MB (68.24%)...
Is there anything I can do?
btw
/home/user> df -i Filesystem Inodes IUsed IFree IUse% Mounted on rootfs 185088 47945 137143 26% / /dev/sda1 185088 47945 137143 26% / unionfs 185088 47945 137143 26% / tmpfs 113339 10 113329 1% /dev/shm tmpfs 113339 47 113292 1% /tmp
Offline
Hi
I think you may have a different issue. Are you getting errors specifically suggesting you have a lack of space?
Our problem was we had few or no iNodes left which meant files could not be created even zero size ones. Your iNode count is fine specifically iFree.
Offline
Sorry for bumping an old-ish thread, but I believe I have a better way to delete those pesky .wh* files.
The method is inspired by this thread, and consists of simply mounting /dev/sda2 in a temp location (like, say, /mnt/sda2), and doing a simple find /temp/location -iname '.wh*' -delete.
Easy-to-follow steps (as commands)
$ su [enter password at prompt] # mkdir -pv /mnt/sda2 # mount /dev/sda2 /mnt/sda2 # find /mnt/sda2 -iname '.wh*' -delete [wait a long-ish while, then check your new IFree count:] # du -i [don't forget to clean up:] # umount /dev/sda2 # exit
UPDATE 2008-07-10: The mount command was missing from this list of commands! This has been corrected now.
I went from ~500-ish free inodes to well over 100k, so...
Methinks a tool to do this should be run periodically -- how quickly the inode table gets filled up seems to depend on how much you use the Eee (assuming the .wh files have something to do with deleting/deleted files, which would make sense as they seem to appear primarily in cache dirs), but it does seem inevitable.
Last edited by Narc (2008-07-10 8:36:34 am)
Offline
If you use sudo su you won't need a password, is there a reason you've used just su?
If you make this as a script could you set up a cron job? (I haven't looked at cron on the Eee, but I know it's installed.)
Last edited by SteveLawUK (2008-06-30 2:59:14 pm)
Offline
SteveLawUK wrote:
If you use sudo su you won't need a password, is there a reason you've used just su?
I used 'su' because I'm lazy and I use it often. You could do it by just prefixing sudo to all commands, and dropping the su altogether.
SteveLawUK wrote:
If you make this as a script could you set up a cron job? (I haven't looked at cron on the Eee, but I know it's installed.)
Here, I'll write your script code for ya:
#!/bin/bash # Create directory if needed (never hurts) sudo mkdir -pv /mnt/sda2 # Mount /dev/sda2 into our new directory sudo mount /dev/sda2 /mnt/sda2 # Delete the stupid .wh* files sudo find /mnt/sda2 -iname '.wh*' -delete # Don't forget to clean up sudo umount /dev/sda2
UPDATE 2008-07-10: The mount command was missing in the previous version of this script! It has now been corrected.
Paste that into a new file, chmod a+x it, then run 'crontab -e' and add this line:
<MIN> <HOUR> <DAY> <MONTH> <DAYOFWEEK> /path/to/cleanup-script
-- replacing:
- <MIN> with a number for the minute the cron job should run (e.g. 17)
- <HOUR> with a number for the hour, or * (asterisk) to run hourly
- <DAY> with a number for the day or * to run daily
- <MONTH> with a number for the month or * to run monthly
- <DAYOFWEEK> with 1-7 representing Mon-Sun or * to run daily
- /path/to/cleanup-script with the path to the script you've created (of course)
By my approximation, running it daily should fix the problem forever, so here's how it would look for running daily at 11:53 am:
53 11 * * * /path/to/the-script
Hope that helps!
ETA: Rant: Considering that, for most applications, you should reach the disk space limit long before the inode limit, and that this whole problem has arisen because of the way the stock Xandros is set up (with UnionFS), this script, or something like it, should have come with the stock install. But, ah, well, at least it gives us geek-types something to fix, yeah?
ETA2: Oh, and cron doesn't seem to be set up to run automatically (at least not on my Eee), so you'll need to set that up, too, somewhere. If you wanted to go the crontab way. Probably easier to just save the script and run it whenever the problem shows up (which should be pretty rarely).
Last edited by Narc (2008-07-10 8:35:28 am)
Offline
Hi Narc
Is it ok to delete all of the .wh files, will any of them be needed? I know most of them are not but they do serve a purpose and I'm wondering if some might actually be valid.
And thanks for the script.
Offline
Nice work on the script too.
Just another comment - I had so many .wh files in the /var/tmp directory, I couldn't just rm ".wh*" because there were so many, it seemed to cause shell commands to fall over! So I had to delete one letter at a time, and ended up writing a little script to do that.
Not sure if "find" is more robust though and able to handle massive numbers of files at once - and anyway, I guess if you ran a cron job as you're suggesting, you'd never build up too many of them at once anyway.
Offline
M1ster_Happy wrote:
Is it ok to delete all of the .wh files, will any of them be needed? I know most of them are not but they do serve a purpose and I'm wondering if some might actually be valid.
I don't honestly know. What I do know is it hasn't seemed to break anything on my Eee, and I've since rebooted it several times.
M1ster_Happy wrote:
And thanks for the script.
No problemo
xMachina wrote:
Just another comment - I had so many .wh files in the /var/tmp directory, I couldn't just rm ".wh*" because there were so many, it seemed to cause shell commands to fall over! So I had to delete one letter at a time, and ended up writing a little script to do that.
Yep, what you ran into is the command line length limit, which is what find(1) (and xargs(1)) were written to address.
xMachina wrote:
Not sure if "find" is more robust though and able to handle massive numbers of files at once
Yep, it is.
xMachina wrote:
and anyway, I guess if you ran a cron job as you're suggesting, you'd never build up too many of them at once anyway.
Yes, but be careful with that, 'cause cron doesn't seem to run by default, as I mentioned.
Offline
Not knowing much about linux, I'd love to get this solution to work for me so I don't have to F9 anymore, but going verbatim off the Narc provided, I'm not getting any changes. I figure this might be due to either the fact I have a lot of free inodes right now (97485) or that I have a 2G surf model, I have no idea. I got the feeling I'm not adding any directory to sda2. I tried the same commanding using sda1 and sdb1, but still no luck. Any ideas on why I'm not able to use this fix or what modifications I might need to get it to work?
Offline
atlantic800 wrote:
Not knowing much about linux, I'd love to get this solution to work for me so I don't have to F9 anymore, but going verbatim off the Narc provided, I'm not getting any changes. I figure this might be due to either the fact I have a lot of free inodes right now (97485) or that I have a 2G surf model, I have no idea. I got the feeling I'm not adding any directory to sda2. I tried the same commanding using sda1 and sdb1, but still no luck. Any ideas on why I'm not able to use this fix or what modifications I might need to get it to work?
The difference is most likely the fact yours is a 2G surf -- I was doing this on a 4G Surf. However, if you have that many free inodes and you're still getting the "No space left on device" error and df -h is showing some free space, I would recommend checking the filesystem.
On my Eee, the F9 menu includes a "Check for errors" entry; if your model does also, try running it, to see if it fixes anything.
Also, can you go to a terminal, run "mount", and copy/paste the output here?
Offline
It's not showing any errors at the moment, just that it has shown the "make sure dcopserver is running error" several times in the past now and I'm hoping to avoid any more F9 recoveries. But like I said, running the commands listed here I don't show any changes to the number of free inodes. I did run the "check for errors" once and it may have prolonged a reinstall, but I had also deleted /var/tmp which is probably what prolonged it.
Here's the output after I run "mount" in terminal
rootfs on / type rootfs (rw)
/dev/sda1 on / type ext2 (ro)
unionfs on / type unionfs (rw,dirs=/=rw:/=ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw)
tmpfs on /dev/shm type tmpfs (rw)
tmpfs on /tmp type tmpfs (rw)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sdb1 on /media/MMC-SD/partition1 type vfat (rw,nosuid,nodev,noexec,fmask=0111,dmask=0000,codepage=cp850,iocharset=utf8,shortname=mixed)
Hey, if nothing else I just learned how to copy/paste out of terminal anyway, but ya, let me know if this gives you any ideas.
Offline
I have tried mounting /dev/sda2 to /mnt/sda2, but it says the device is in use (which it is, by the unionfs thingy) is there any way to force the mount?
The script above does not have a mount in it, only a mkdir, then a find on an empty folder... this obviously does not work.
The f9 busybox console does not have find, so I can't do it there either.
Offline
atlantic800 wrote:
[...]But like I said, running the commands listed here I don't show any changes to the number of free inodes.[...]
Actually, that's my fault. See below for corrected script.
beckylou wrote:
I have tried mounting /dev/sda2 to /mnt/sda2, but it says the device is in use (which it is, by the unionfs thingy) is there any way to force the mount?
Hm... if there is, I don't know about it, sadly. However, the odd thing is it mounts just fine for me. This points to a bit more variation than is immediately apparent.
beckylou wrote:
The script above does not have a mount in it, only a mkdir, then a find on an empty folder... this obviously does not work.
Yeah, that's my mistake; I'm surprised nobody else noticed it Corrected version for everyone else:
#!/bin/bash # Create directory if needed (never hurts) sudo mkdir -pv /mnt/sda2 # Mount /dev/sda2 into our new directory sudo mount /dev/sda2 /mnt/sda2 # Delete the stupid .wh* files sudo find /mnt/sda2 -iname '.wh*' -delete # Don't forget to clean up sudo umount /dev/sda2
I'll correct it in the earlier post, too.
beckylou wrote:
The f9 busybox console does not have find, so I can't do it there either.
At this point, the only thing I can think of is grabbing something like PuppyLinux or another small distro, setting it up on a USB flash drive, and booting it, then using it to mount sda2 and do the cleanup.
Offline
Don't the 900s use sdb for the user partition?
Offline
beckylou wrote:
I have tried mounting /dev/sda2 to /mnt/sda2, but it says the device is in use (which it is, by the unionfs thingy) is there any way to force the mount?
I mounted it successfully to /tmp/sda2 (a ram disk) - i think it was an inode problem rather than a mount problem.
Also, with regards to the missing crontab thing - you can call the script from /usr/sbin/services.sh - then it'll get run on every bootup
Offline
Thanks a lot everybody
I ended up incorporating beckylou's /tmp solution into Narc's commands and so I ran
sudo mkdir -pv /tmp/sda2
sudo mount /dev/sda2 /tmp/sda2
sudo find /tmp/sda2 -iname '.wh*' -delete
sudo umount /dev/sda2
which left me with only 1314 used inodes
Honestly I can't say I understand any of it, so I don't know what the last command accomplishes for me, if anything. I'll have to make an effort to read up on mounting. If anyone comes up with a detailed description for writing and incorporating a little program that runs automatically every week I would be ecstatic. If not I'll see if I can't make one myself, I know all the real info is already in this thread. Thanks a lot for the help all, no more unwanted F9 reinstalls! Free inodes for all!
Offline
^ Last command unmounts the filesystem so you can't access it through the directory /tmp/sda2 anymore. At that point, you can also rmdir /tmp/sda2.
Glad my solution worked for you, too
ETA: Also, excellent suggestion to use /tmp, beckylou!
Last edited by Narc (2008-07-10 2:19:26 pm)
Offline
The main problem is that UnionFS is not checking partition1 to see if it *needs* to make a .wh file - it would be a million times better if it did this because it wouldn't need to make the .wh files on every delete of any file.
A lot of applications will open a file handle to the disk, put some data in it (kept in file-cache without writing it to the disk) and then discard the file and delete it. Mainly because it allows you to have a middle ground between "as big as you want" and "as fast as possible" - with big data sets it'll go over the cache buffer and write to the disk, but small data sets will remain in memory and be really quick. The down side is that on SSD this hammers the wearleveling, and unionFS even more-so, because it write those cursed .wh files!
I am looking into getting SQLite temp files to write to /tmp instead of /var/tmp - this is one of the main users of inodes.
Offline
beckylou wrote:
[...]I am looking into getting SQLite temp files to write to /tmp instead of /var/tmp - this is one of the main users of inodes.
Me, I'm curious if /var/tmp can't just be made into a tmpfs. I'm not curious enough to try it yet, though.
I suspect it'll require write access to /dev/sda1, though.
Offline
No, /var/tmp is for semi-perminant files that need to persist across reboots, putting them on a ramdisk would be a bad idea
Offline
Does anyone see any problems with running these commands frequently? I just put them into the commands to run when the lid is closed, so it will run whenever my computer is on and I shut the lid. Guess I'll report it back on here if this causes a problem.
Offline
Pages: 1 2
