Double-guessing the user's intention is so reliable. Maybe I really do mean m -rfy /

There's an amazing Github project called shellcheck that already does a lot of this work... I wonder if the researchers were aware of it?

I have a feeling the researchers weren't interested in existing solutions, instead they had AI to peddle. From TFA:

Using large language models to check shell command documentation against actual behavior

Great, we can look forward to correctness when compared against some bash/ksh combined AI slop scripting language.

Shell programming checks - apply Occam's razor.

I'll clam up now.

So did Crowdstrike, that did not end well. ;)

Surely this is a bag of a fag packet task for "AI" these days ?

" bag of a fag packet task "

Sometime a typo just makes it golden. :))

I'll wait 10 years for the really hot one

1) The first line should be "#! /bin/bash -e". That -e means exit on error. If the author missed it then when the script finds something unexpected it will plow on regardless and do things you do not want. This tells you enough about the author to not run his scripts.

2) If the script is over 50 lines long it should have been written in a proper high level language.

Nobody expects the Spanish inquisition. Our three tests are 1) -e, 2) script length, and...

3) bash -n - syntax check the script

4) set -u, to make using an undeclared variable an error (good for catching typos like FILENAME=$(foo) ; rm -f $FILENME)

Worse:

DIRNAME=TMP

rm -rf /${DIRNAM}

Yes, there are (multiple) things programmers can do to prevent the bad results from this type of error. Despite that, someone at vALVE made this sort of error with ~/ in a script that was pushed to all on-line STEAM users some years back, wiping out all the affected users' stats and saved games.

(Luckily for me, my STEAM-running games PC was powered off that week.)

I agree that the exit-on-error & error-on-undefined flags should always be set either via the shebang or the set command at the top of the file.

If using bash, then set -o pipefail is also a good idea.

But the first line should be #!/bin/sh -e not #!/bin/bash -e unless the script is actually using bash features like shopt -s lastpipe , arrays, local variables etc.

Using bash for shell scripts that only need basic POSIX shell features wastes a lot of resources, especially if said script is going to be run repeatedly from cron or via other background triggering.

On 64-bit Raspbian for example:-

$ ls -l /bin/{ba,da,}sh

-rwxr-xr-x 1 root root 1346480 Mar 29 2024 /bin/bash

-rwxr-xr-x 1 root root 133640 Jan 5 2023 /bin/dash

lrwxrwxrwx 1 root root 4 Jan 5 2023 /bin/sh -> dash

$ ls -l /bin/bash

ls: /bin/bash: No such file or directory

If you must use bash features it should be launched with #!/usr/bin/env bash . /usr/bin/env is almost standardised.

Moreover set -e is overrated and unreliable. If a command can fail the script should check for the failure explicitly.

Unix, which is what's really being programmed here the shell language is a distraction, is user friendly: it's just picky about who its friends are.

I always go for bash first because there is always some utility that is quicker to write in bash than some other higher level language.

- It's always : #!/usr/bin/env bash

- set -euo pipefail (or set -eo pipefail, if the script is not destructive but if not -u then we do...

- local varname=${some_otther_varname:?undefined variable, abort}

Because the :? is a very useful bash variable expansion...

You can do some awful awful things with a combination of tools like yq + jq and still store your "configuration" in semi-readable YAML.

I know it works and is the more portable thing to do but #!/usr/bin/env bash seems illogical from a purely abstract point of view.

Avoiding an absolute path for invoking one program via the absolute path to another seems daft!

Anyway, back on topic, some other ways of improving code quality is always double-quoting substitutions (completely e.g. cmd1 "$(cmd2 "${var}")" ), never using backticks for command substitutions, using xargs or mapfile instead of command substitution.

As someone said, installing shellcheck to lint all your scripts is a good place to start and will pick up these and many other potential gotchas.

Moreover set -e is overrated and unreliable. If a command can fail the script should check for the failure explicitly.

I agree that if the failure of a command would lead to the script leaving things in an inconsistent state, then it should be explicitly invoked via if or by checking $? .

But checking the result of every external command, shell built-in & shell function that might fail is exhausting if a simple abort of the script - maybe with a trap EXIT to clean up - is less error prone and in any case set -e will act as a last resort assertion.

I take some kind of morbid pride in the horrendous shell scripts that litter the landscape of many a company I've worked for over the past 30 years. If your eyes don't bleed, you've done it wrong.

I find the very concept of checking a shell script is a hiding to nothing.

The issue is that very little of what is written in a shell script is actually written in shell commands. A huge amount is just herding other (external) commands to get a task done, passing data through pipelines of non-shell tools to achieve the result.

As such, I like to think of the shell (pretty much any variant since UNIX Edition 6 shell, which predates the Bourne shell) as more of a harness for holding things together with some programming structures to make life easier, rather than a fully specified programming language in it's own right. In addition, it has handling of wild-card and variable substitution that make it very suitable for tasks that would be very difficult to code in a more formal language.

There are two reasons I think this. A while ago, some article or other issued a challenge to write some shell to do something. I can't remember exactly what it was, but it was date related. The majority of posted solutions were posted as shell scripts, but in which, most of the actual processing was not done using shell built-ins. They were shell commands calling other tools like cal, or date, and using something like awk, sed and various other tools to process the output to produce the required result.

I actually tried to write a version that was pure shell. And it was very difficult. I was working in ksh88, but I also wrote versions in ksh93. This is not what shells were written to do!

The second reason is that I have been recently reviewing some Python which is being used to run Ansible playbooks, and this often involves quite a few hoops to run external command required for automation tasks, and then process the results in Python to work out how well it worked or whether it failed. I look at these programs, and then imagine what I would do in a shell, and find that quite often it would be much, much easier to do it as a series of commands in something like a pipeline, run from shell, rather than trying to do it in Python. For my own tasks that I have been told I need to write, I'm really thinking of writing shell scripts together with a deploy method in ansible, and playbooks that just call the shell scripts. It would be much easier to write, and (IMHO) will be easier to maintain, although it's obviously subverting the whole reasoning of Ansible.

In conclusion, what I think I'm trying to say is that the way shell and shell scripts are used should not be treated as a formal language, and as such is very difficult to subject to automated code review.

I recall the Plan9 shell, rc was a more modern design intended to address some of the shortcomings of the Bourne and Korn shells ([t]csh wasn't used for serious scripting by anyone who valued their sanity) and bash either didn't exist or was early days.

Years ago I had rc running on a hpux workstation with a X11 port of the window manager 9wm(?) which as I recall was sort of ok.

I know I knocked out a colossal quantity of shell script mostly to glue unrelated applications together

Awk and Perl always seemed to have a impedance mismatch with the OS and file system which just made shell seem an easier choice.

Until the ascendance of Linux distributions the variety of Unix meant that nothing much beyond Bourne shell could be assumed. Even awk was old awk, perl ancient or not present, definitely no python by default. The init script on hpux 10.20 that configured (multiple) network interfaces and aliases was an impenetrable wonder of Bourne shell gymnastics.

Part of the problem, I suspect, is that users would prefer their scripting language to be the same as their interactive shell which probably present inreconcilable design goals.

To tell you the truth, for all I mistrust it, I suspect that something like PowerShell will end up being the way forward. But in using this, you need an OS whose components can and do understand complex data objects.

I actually quite liked the VM/CMS implementation of REXX as a command processing language, but the OS/2 and AIX implementations were not complete, so never really gained any headway.

One of the biggest problems (and past strengths as well) is that UNIX-like commands are designed to work on streams of bytes, often arranged as lines. This made every sense when OS's were more simple than they have to be now, and commands were written to be run as interactive commands. But modern problems are often much more complex than can be represented by just a stream of bytes.

If an OS's command set was re-implemented to use some form of object passing, this would make much of the convoluted shell scripting that has been required in the past redundant.

But it would no longer really be a UNIX-like OS!

Boy, am I glad I'm retiring shortly.