A start job is running for Create Volatile Files and Directories

Posted on Edited on In 评论: Views:

Yesterday, when I login to my VM I found that the disk is full. Even a tab command completion cannot be done. I immediately delete some unused files and the system works. However, when I reboot the system, it stucked at A start job is running for Create Volatile Files and Directories...:

...
[  OK  ] Started File System Check on /dev/disk/cloud/azure_resource-part1.
[  OK  ] Started File System Check Daemon to report status.
[  OK  ] Started File System Check on /dev/d…5d996-7436-4ec8-b5d6-0d7b6100aeb5.
         Mounting /data...
[  OK  ] Mounted /data.
[  OK  ] Reached target Local File Systems.
         Starting AppArmor initialization...
         Starting ebtables ruleset management...
         Starting Tell Plymouth To Write Out Runtime Data...
         Starting Set console font and keymap...
[  OK  ] Started Set console font and keymap.
[  OK  ] Started Tell Plymouth To Write Out Runtime Data.
[  OK  ] Started ebtables ruleset management.
[  OK  ] Started Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Started AppArmor initialization.
[    **] A start job is running for Create V… Directories (13min 46s / no limit)

I searched and found a similar discussion: # Boot stuck at “A start job is running for Create Volatile Files and Directories”

I used an Azure VM with an extra disk. However, it always failed to boot into recovery mode. Then I removed the extra disk from the portal and the system mount the disk failed and went to the single user mode so that I can execute commands:

root@DEAN:/tmp# ls /tmp
bash: /bin/ls: Argument list too long

Then occasionally I found that there are HUGE amount of something like this .com.google.Chrome.nFbq9T: 

root@DEAN:/tmp# ls .com.google.Chrome.*
bash: /bin/ls: Argument list too long

Since I have a web crawler in the system, it seems that there are something wrong but it doesn't matter. I also cannot execute rm -f .com.google.Chrome.* due to Argument list too long. First list them:

root@DEAN:/home# find /tmp -type f -iname ".com.google.Chrome*" -ls | head
find: ‘/tmp/.com.google.Chrome.nFbq9T’: No such file or directory
find: ‘/tmp/.com.google.Chrome.WKCRCU’: No such file or directory
find: ‘/tmp/.com.google.Chrome.62cCbn’: No such file or directory
find: ‘/tmp/.com.google.Chrome.RR1E3L’: No such file or directory

The command cannot quickly return, but I got above error messages. Seems that the OS is cleaning the /tmp folder but I cannot verify this. Through df -lh I found that the available disk space is increasing. To speed up the process, execute:

root@DEAN:/home# find /tmp -iname ".com.google.Chrome*" -delete > /dev/null 2>&1

After nearly an hour it finished: 

~$ df -lh
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           393M  1.1M  392M   1% /run
/dev/sda1        29G   13G   16G  46% /
...

It looks like there were 16G .com.google.Chrome. files... And because of this, the system is stuck at boot. Unfortunately, the root cause of this problem is not found yet.

By the way, directly move /tmp to /tmp.bak and recreate /tmp might be more efficient as suggested in the above post...