Surprising shell pathname expansion

I thought I was understanding pretty well how bash argument processing and various expansions is supposed to behave. Apparently, there are still subtleties which tricks me, sometimes.

Question: what is the (standard) output of the following shell command? 🤔

a='*' ; echo $a

The answer is below this anti-spoiler protection.

Answer

Here is the command again:

a='*' ; echo $a

I would have said that the answer was *, obviously. But this is wrong. The output is the list of files in the current directory. 😲

The content of the a variable is * because the assignment is single-quoted. For example, this shell command does output *:

a='*' ; echo "$a"

However, in echo $a, * is pathname-expanded into the list of files in the current directory. I would not have thought that pathname expansion would trigger in this case.

Explanation

This is indeed the behaviour specified for POSIX shell Word Expansions:

The order of word expansion shall be as follows:

1. Tilde expansion (see Tilde Expansion), parameter expansion (see Parameter Expansion), command substitution (see Command Substitution), and arithmetic expansion (see Arithmetic Expansion) shall be performed, beginning to end. See item 5 in Token Recognition.
2. Field splitting (see Field Splitting) shall be performed on the portions of the fields generated by step 1, unless IFS is null.
3. Pathname expansion (see Pathname Expansion) shall be performed, unless set -f is in effect.
4. Quote removal (see Quote Removal) shall always be performed last.

Pathname expansions happens after variable expansion. I think I would have said it was done before variable expansion and command substitution.

Edit: I think what I actually found surprising is that pattern matching characters coming from expansions are actually active pattern matching characters (instead of counting as ordinary characters).

About Parameter Expansion POSIX mandates that double-quotes prevents pathanme expansions from happening (i.e. if there is no quoting pahtname expansion happens):

If a parameter expansion occurs inside double-quotes: Pathname expansion shall not be performed on the results of the expansion.

Of course, single quotes prevents pathname expansion from happening (in addition to preventing variable expansion and otherthings from happening):

Enclosing characters in single-quotes ('') shall preserve the literal value of each character within the single-quotes. A single-quote cannot occur within single-quotes.

This is not super surprising if we think about, for example:

# List all HTML files:
ext=html ; echo *.$ext

This works as well with pattern matching:

ext=html
for a in "$@"; do
  case "$a" in
    *.$ext)
        echo "Interesting file: $a"
        ;;
    *)
      echo "Boring file: $a"
      ;;
  esac
done

Command Substitution

About Command Substitution POSIX mandates:

If a command substitution occurs inside double-quotes, field splitting and pathname expansion shall not be performed on the results of the substitution.

Which means that this command outputs the list of file in the current directory as well:

echo $(echo '*')

Context

It took me some time to understand what was happening when debugging a slightly more convoluted example from YunoHost:

ynh_mysql_execute_as_root "GRANT ALL PRIVILEGES ON *.* TO '$db_admin_user'@'localhost' IDENTIFIED BY '$db_admin_pwd' WITH GRANT OPTION;
  FLUSH PRIVILEGES;" mysql

Inside ynh_mysql_execute_as_root, the parameters are assigned to local variables with this (bash) code:

arguments[$i]="${arguments[$i]//\"/\\\"}"
arguments[$i]="${arguments[$i]//$/\\\$}"
eval ${option_var}+=\"${arguments[$i]}\"

This code is obviously vulnerable to shell command code injection in the eval line through backticks and backslashes. What surprised me is that pathname expansion was happening in *.*. This is because ${arguments[$i]} is not double-quoted in the last line and this is completely unrelated to eval.

For reference, the correct and simple way to proceed, which avoids unwanted command injection and pathname expansion is:

eval ${option_var}+='"${arguments[$i]}"'

Conclusion

Unquoted variable expansion and command substitutions are trickier than I thought.

When variable expansion or command substitution happens unquoted, pathname expansion might possibly happen. I think this might have security implications for some shell scripts out there.