41 replies to this topic

[xMachina]

Hi,

I've had some major troubles with my little EEE's filesystem in the past few days, have searched the forums quite a bit but haven't found an answer to the problem yet.

I think my long-term strategy is to remove the unionfs and go with straight ext2, but it's the last week of Uni this week so if possible would love to find a "quick fix" until I have time to do a proper upgrade in the holidays!

The sympoms are these: a few days ago, I booted my EEE-PC and things like Wireless failed to start and gave error messages. Investigating in the terminal I find the cause seems to be "no space left on device", e.g. trying "/etc/init.d/networking restart" eventually fails with a message "cannot touch '/etc/hosts.new': No space left on device".

THis is happening despite the output of df -h which is:
Filesystem Size Used Avail Use% Mounted on
rootfs 1.4G 727M 623M 54% /
/dev/sda1 1.4G 727M 623M 54% /
unionfs 1.4G 727M 623M 54% /
tmpfs 249M 4.0K 249M 1% /dev/shm
tmpfs 128M 12K 128M 1% /tmp
/dev/sdhc1 7.5G 707M 6.5G 10% /home/user/homeSDHC
/dev/sdc1 3.9G 215M 3.6G 6% /media/Cruzer Crossfire

and typing mount gives the following which looks pretty normal:
rootfs on / type rootfs (rw)
/dev/sda1 on / type ext2 (ro)
unionfs on / type unionfs (rw,dirs=/=rw:/=ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw)
tmpfs on /dev/shm type tmpfs (rw)
tmpfs on /tmp type tmpfs (rw)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sdhc1 on /home/user/homeSDHC type ext2 (rw)
/dev/sdc1 on /media/Cruzer Crossfire type vfat (rw,fmask=0022,dmask=0022,codepage=cp850,iocharset=iso8859-1)

I recently tried booting up using a "TRK rescue" Linux boot USB disk I created a while back, and deleting some non-essential files off the "protected" HDD (eg /usr/share/games), but it didn't seem to help.

Any ideas/help greatly appreciated! ;)

[M1ster_Happy]

Hi

I've just had a similar problem. For me it turned out to be the iNodes.... do 'df -i' and see if you have zero or low number of IFree. I was on between zero and 20. I found out lots of zero size .wh.? files are created when you delete things for UnionFS I think. I ended up doing what you did and booting off of Parted Magic and deleted some .wh files as it was too unstable to use file manager (9000 in my opera cache directory).... still in the process of deleting now but running a lot smoother.

Hope this helps ... oh i did a 'ls -alR |more' to give me some idea where there was lots of similar files.

=)

Edited by M1ster_Happy, 03 June 2008 - 05:50 AM.

[xMachina]

Thanks M1ster_Happy, you are a champ! Yes, Inodes was down to zero, and as you suggested lots of .wh. files was a big culprit- for me it was mainly in the /var/tmp directory, lots of sqlite files it seems.

Surprising that more people haven't run into this problem?? Maybe we should create a Wiki entry.

Oh well, it's great to be up & running again, & reinforces my plan to get rid of unionfs when I have more tinkering time.

[M1ster_Happy]

Yeah I just found the tmp ones after the post .... there was ... alot! I was hoping the files would be visible when booting normally but it seems I have to boot off of Parted Magic or another live version to see/delete them.

To be honest I wouldn't want to recommend it in a wiki as I'm a total linux newbie so it may turn out to be wrong! (I hope not) For now if you can change the thread name you could prefix SOLVED on it, just an idea.

I'm not really sure why they get created as they seem to be a 'white out' file to hide the deleted file on the lower partition of the unionfs set up... but they get created in directories that don't exist on the lower partition, and in directories that have '.wh.__dir_opaque' files which is suggesting lower partition directories to be ignored anyway (so i've read.. maybe wrong! ). I'm sure there's some logic to it but the way it seems it should be fine to delete any '.wh.*' files where a '.wh.__dir_opaque' file exists (not deleting the opaque), and where the lower partitions directory is empty or does not exist. I think you have to watch out for deleting a deleted directory file as well which is 'hiding' the directory in existence on the lower partition (don't think its too much of a problem as it will just be visible again).

I'd like to write a script to do it but like you I don't believe I need the unionfs system so I think I will just remove it.... but until then it seems deleting them will keep us going. Like you said you would think more people would have the same issue... but then alot of mine were opera so i guess I wouldn't have hit the issue for a while longer... guess we'll see.

Let me know if you get any issues.

=)

[CCG]

Hmmm I have found similar stuff overhere:

/home/user> df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                1.4G  452M  898M  34% /
/dev/sda1             1.4G  452M  898M  34% /
unionfs               1.4G  452M  898M  34% /
tmpfs                 443M   16K  443M   1% /dev/shm
tmpfs                 128M   56K  128M   1% /tmp

While diskutil shows use 451MB (31.76%) and available 969MB (68.24%)...

Is there anything I can do?

btw

/home/user> df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
rootfs                185088   47945  137143   26% /
/dev/sda1             185088   47945  137143   26% /
unionfs               185088   47945  137143   26% /
tmpfs                 113339      10  113329    1% /dev/shm
tmpfs                 113339      47  113292    1% /tmp

[M1ster_Happy]

Hi

I think you may have a different issue. Are you getting errors specifically suggesting you have a lack of space?

Our problem was we had few or no iNodes left which meant files could not be created even zero size ones. Your iNode count is fine specifically iFree.

=)

[Narc]

Sorry for bumping an old-ish thread, but I believe I have a better way to delete those pesky .wh* files.

The method is inspired by this thread, and consists of simply mounting /dev/sda2 in a temp location (like, say, /mnt/sda2), and doing a simple find /temp/location -iname '.wh*' -delete.

Easy-to-follow steps (as commands)

$ su
[enter password at prompt]
# mkdir -pv /mnt/sda2
# mount /dev/sda2 /mnt/sda2
# find /mnt/sda2 -iname '.wh*' -delete
[wait a long-ish while, then check your new IFree count:]
# du -i
[don't forget to clean up:]
# umount /dev/sda2
# exit

UPDATE 2008-07-10: The mount command was missing from this list of commands! This has been corrected now.

I went from ~500-ish free inodes to well over 100k, so...

Methinks a tool to do this should be run periodically -- how quickly the inode table gets filled up seems to depend on how much you use the Eee (assuming the .wh files have something to do with deleting/deleted files, which would make sense as they seem to appear primarily in cache dirs), but it does seem inevitable.

Edited by Narc, 10 July 2008 - 12:36 PM.

[SteveLawUK]

If you use sudo su you won't need a password, is there a reason you've used just su?

If you make this as a script could you set up a cron job? (I haven't looked at cron on the Eee, but I know it's installed.)

Edited by SteveLawUK, 30 June 2008 - 06:59 PM.

[Narc]

Quote

If you use sudo su you won't need a password, is there a reason you've used just su?

I used 'su' because I'm lazy and I use it often. You could do it by just prefixing sudo to all commands, and dropping the su altogether.

Quote

If you make this as a script could you set up a cron job? (I haven't looked at cron on the Eee, but I know it's installed.)

Here, I'll write your script code for ya:

#!/bin/bash

# Create directory if needed (never hurts)
sudo mkdir -pv /mnt/sda2

# Mount /dev/sda2 into our new directory
sudo mount /dev/sda2 /mnt/sda2

# Delete the stupid .wh* files
sudo find /mnt/sda2 -iname '.wh*' -delete

# Don't forget to clean up
sudo umount /dev/sda2

UPDATE 2008-07-10: The mount command was missing in the previous version of this script! It has now been corrected.

Paste that into a new file, chmod a+x it, then run 'crontab -e' and add this line:

<MIN> <HOUR> <DAY> <MONTH> <DAYOFWEEK> /path/to/cleanup-script

-- replacing:
- <MIN> with a number for the minute the cron job should run (e.g. 17)
- <HOUR> with a number for the hour, or * (asterisk) to run hourly
- <DAY> with a number for the day or * to run daily
- <MONTH> with a number for the month or * to run monthly
- <DAYOFWEEK> with 1-7 representing Mon-Sun or * to run daily
- /path/to/cleanup-script with the path to the script you've created (of course)

By my approximation, running it daily should fix the problem forever, so here's how it would look for running daily at 11:53 am:

53 11 * * * /path/to/the-script

Hope that helps!

ETA: Rant: Considering that, for most applications, you should reach the disk space limit long before the inode limit, and that this whole problem has arisen because of the way the stock Xandros is set up (with UnionFS), this script, or something like it, should have come with the stock install. But, ah, well, at least it gives us geek-types something to fix, yeah? :)

ETA2: Oh, and cron doesn't seem to be set up to run automatically (at least not on my Eee), so you'll need to set that up, too, somewhere. If you wanted to go the crontab way. Probably easier to just save the script and run it whenever the problem shows up (which should be pretty rarely).

Edited by Narc, 10 July 2008 - 12:35 PM.

[M1ster_Happy]

Hi Narc

Is it ok to delete all of the .wh files, will any of them be needed? I know most of them are not but they do serve a purpose and I'm wondering if some might actually be valid.

And thanks for the script.

=)

[xMachina]

Nice work on the script too.

Just another comment - I had so many .wh files in the /var/tmp directory, I couldn't just rm ".wh*" because there were so many, it seemed to cause shell commands to fall over! So I had to delete one letter at a time, and ended up writing a little script to do that.

Not sure if "find" is more robust though and able to handle massive numbers of files at once - and anyway, I guess if you ran a cron job as you're suggesting, you'd never build up too many of them at once anyway.

[Narc]

Quote

Is it ok to delete all of the .wh files, will any of them be needed? I know most of them are not but they do serve a purpose and I'm wondering if some might actually be valid.

I don't honestly know. What I do know is it hasn't seemed to break anything on my Eee, and I've since rebooted it several times.

Quote

And thanks for the script.

No problemo :)

Quote

Just another comment - I had so many .wh files in the /var/tmp directory, I couldn't just rm ".wh*" because there were so many, it seemed to cause shell commands to fall over! So I had to delete one letter at a time, and ended up writing a little script to do that.

Yep, what you ran into is the command line length limit, which is what find(1) (and xargs(1)) were written to address.

Quote

Not sure if "find" is more robust though and able to handle massive numbers of files at once

Yep, it is.

Quote

and anyway, I guess if you ran a cron job as you're suggesting, you'd never build up too many of them at once anyway.

Yes, but be careful with that, 'cause cron doesn't seem to run by default, as I mentioned.

[atlantic800]

Not knowing much about linux, I'd love to get this solution to work for me so I don't have to F9 anymore, but going verbatim off the Narc provided, I'm not getting any changes. I figure this might be due to either the fact I have a lot of free inodes right now (97485) or that I have a 2G surf model, I have no idea. I got the feeling I'm not adding any directory to sda2. I tried the same commanding using sda1 and sdb1, but still no luck. Any ideas on why I'm not able to use this fix or what modifications I might need to get it to work?

[Narc]

Quote

Not knowing much about linux, I'd love to get this solution to work for me so I don't have to F9 anymore, but going verbatim off the Narc provided, I'm not getting any changes. I figure this might be due to either the fact I have a lot of free inodes right now (97485) or that I have a 2G surf model, I have no idea. I got the feeling I'm not adding any directory to sda2. I tried the same commanding using sda1 and sdb1, but still no luck. Any ideas on why I'm not able to use this fix or what modifications I might need to get it to work?

The difference is most likely the fact yours is a 2G surf -- I was doing this on a 4G Surf. However, if you have that many free inodes and you're still getting the "No space left on device" error and df -h is showing some free space, I would recommend checking the filesystem.

On my Eee, the F9 menu includes a "Check for errors" entry; if your model does also, try running it, to see if it fixes anything.

Also, can you go to a terminal, run "mount", and copy/paste the output here?

[atlantic800]

It's not showing any errors at the moment, just that it has shown the "make sure dcopserver is running error" several times in the past now and I'm hoping to avoid any more F9 recoveries. But like I said, running the commands listed here I don't show any changes to the number of free inodes. I did run the "check for errors" once and it may have prolonged a reinstall, but I had also deleted /var/tmp which is probably what prolonged it.
Here's the output after I run "mount" in terminal

rootfs on / type rootfs (rw)
/dev/sda1 on / type ext2 (ro)
unionfs on / type unionfs (rw,dirs=/=rw:/=ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw)
tmpfs on /dev/shm type tmpfs (rw)
tmpfs on /tmp type tmpfs (rw)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sdb1 on /media/MMC-SD/partition1 type vfat (rw,nosuid,nodev,noexec,fmask=0111,dmask=0000,codepage=cp850,iocharset=utf8,shortname=mixed)

Hey, if nothing else I just learned how to copy/paste out of terminal anyway, but ya, let me know if this gives you any ideas.

[beckylou]

I have tried mounting /dev/sda2 to /mnt/sda2, but it says the device is in use (which it is, by the unionfs thingy) is there any way to force the mount?

The script above does not have a mount in it, only a mkdir, then a find on an empty folder... this obviously does not work.

The f9 busybox console does not have find, so I can't do it there either.

[Narc]

Quote

[...]But like I said, running the commands listed here I don't show any changes to the number of free inodes.[...]

Actually, that's my fault. See below for corrected script.

Quote

I have tried mounting /dev/sda2 to /mnt/sda2, but it says the device is in use (which it is, by the unionfs thingy) is there any way to force the mount?

Hm... if there is, I don't know about it, sadly. However, the odd thing is it mounts just fine for me. This points to a bit more variation than is immediately apparent.

Quote

The script above does not have a mount in it, only a mkdir, then a find on an empty folder... this obviously does not work.

Yeah, that's my mistake; I'm surprised nobody else noticed it :) Corrected version for everyone else:

#!/bin/bash

# Create directory if needed (never hurts)
sudo mkdir -pv /mnt/sda2

# Mount /dev/sda2 into our new directory
sudo mount /dev/sda2 /mnt/sda2

# Delete the stupid .wh* files
sudo find /mnt/sda2 -iname '.wh*' -delete

# Don't forget to clean up
sudo umount /dev/sda2

I'll correct it in the earlier post, too.

Quote

The f9 busybox console does not have find, so I can't do it there either.

At this point, the only thing I can think of is grabbing something like PuppyLinux or another small distro, setting it up on a USB flash drive, and booting it, then using it to mount sda2 and do the cleanup.

[SteveLawUK]

Don't the 900s use sdb for the user partition?

[beckylou]

Quote

I have tried mounting /dev/sda2 to /mnt/sda2, but it says the device is in use (which it is, by the unionfs thingy) is there any way to force the mount?

I mounted it successfully to /tmp/sda2 (a ram disk) - i think it was an inode problem rather than a mount problem.

Also, with regards to the missing crontab thing - you can call the script from /usr/sbin/services.sh - then it'll get run on every bootup :)

[atlantic800]

Thanks a lot everybody
I ended up incorporating beckylou's /tmp solution into Narc's commands and so I ran

sudo mkdir -pv /tmp/sda2

sudo mount /dev/sda2 /tmp/sda2

sudo find /tmp/sda2 -iname '.wh*' -delete

sudo umount /dev/sda2

which left me with only 1314 used inodes

Honestly I can't say I understand any of it, so I don't know what the last command accomplishes for me, if anything. I'll have to make an effort to read up on mounting. If anyone comes up with a detailed description for writing and incorporating a little program that runs automatically every week I would be ecstatic. If not I'll see if I can't make one myself, I know all the real info is already in this thread. Thanks a lot for the help all, no more unwanted F9 reinstalls! Free inodes for all!