Today I encounter a very strange problem that all recent deployed applications in a specific host fail to start, with a simple error message “Permission Denied. xxxx.sh could not be executed”.
Nonsense! They’ve run for a very long time with a Jenkins driven deployment, started by a deploy script. And a binary “could not be executed” might be controlled by user & file access flag. Both checked, using
root user &
755 access flag.
Suddenly I remember that the hard-disk on this host have been re-mounted by Ops a few days ago, this operation might corrupt some filesystem runtime context. So I wrote a simple test script
a.sh, place it into different mounting point
/data and try to execute them via
bash ./a.sh because in this case we are using
r way instead of
x to access the script). And script could be executed under
/tmp but not
Seems we are close to the root cause, but what stop us from executing a binary in different mounting point? The answer is the
fstab is a configuration file that contains information of all the partitions and storage devices in your computer. The file is located under /etc, so the full path to this file is /etc/fstab. /etc/fstab contains information of where your partitions and storage devices should be mounted and how.
After viewing the content of
/etc/fstab, we then know why things happen. Here are the content:
There are 6 columns for a mount option, each represent:
- block special device or remote filesystem to be mounted
- mount point of file system
- file system type
- mount option associate with the mount
- dump(8) flag
- boot check sequence
With help of
man 5 fstab and
man 8 mount we could see the
/data mounting point is bound with a ridiculous
noexec option. According to man page of
fstab(5) about fourth field (fs_mntops):
- This field describes the mount options associated with the filesystem.
- It is formatted as a comma separated list of options.
- It contains at least the type of mount plus any additional options appropriate to the filesystem type.
- For documentation on the available mount options, see mount(8).
- For documentation on the available swap options, see swapon(8).
- noexec Do not allow direct execution of any binaries on the mounted filesystem. (Until recently it was possible to run binaries anyway using a command like /lib/ld*.so /mnt/binary. This trick fails since Linux 2.4.25 / 2.6.0.)
So problem resolved after we remove the “,noexec” option from the
/data mount point. The previous statement “(Binary) could not be executed might be controlled by user & file access flag.” are not accurate enough. Binary execution could also be controlled by File system mount options under “/etc/fstab”.
Now it’s time to ask why Ops assigned such a flag on this mount…