You should leave off the square brackets when you are testing the exit code of a function, otherwise, you ll always get "true". Also, you should quote your variable. You could put an additional test for an empty extension as shown:

FILE_EXT=${FILE#*.}

if isNumeric "${FILE_EXT}" &&
    [ "${FILE_EXT}" != "${FILE}" -a "${FILE_EXT}" != "" ]
then
    echo "Numbered file."
fi

Edit: added test to handle filenames that end in "."

I would do this:

if [ isNumeric ${FILE_EXT:="no"} ]; then
    echo "Numbered file."
fi

If all you want to do is determine that the file has a numeric extension

The ${FILE_EXT:="no"} will expand to either the value of FILE_EXT, or no if FILE_EXT is null or empty.

you should use ${FILE##*.} with double "#" instead. also what do you mean the variable $FILE_EXT will be empty? if your file don t have extension, then when you do ${FILE#*.} you will get the just file name in FILE_EXT. how is it empty ?

Assuming ksh93, it should be possible to use it s own arithmetic. But we need to be careful: Just ((n)) will fail if n==0, so we test for ((n || !n)) which should always be true for any proper number.

To prevent ksh from exiting, we run the expression in a subshell ( ), adding spaces to prevent conflicts with the arithmetic expression (( ... )).

Finally, we close stderr with 2>& - to prevent any error messages from non numeric arguments, although you might want to keep them in.

function isNumeric { 
  (
    typeset n=${1:?}
    ((n||!n))
  ) 2>& -
}

[ -z "$1" ] will test for an empty $1, as will [ "" = "$1" ].

Or you could simply prepend a "0" to $1 (i.e, "0$1") to force it to be non-empty before checking if it is numeric (if you want empty extensions to be treated as numeric).

I had some probs to run your script (Maybe that s because im using pdksh). So I have adjusted it slightly. Try this:

#!/usr/bin/ksh

FILE=$1
FAILURE=1
SUCCESS=0

isNumeric ()
{
    if [ -n "$1" ]
    then
        case $1 in
            *[!0-9]* | "")
                echo "$1 not a number"
                return $FAILURE
            ;;
                       * )
                echo "$1 is a number"
                return $SUCCESS
            ;;
        esac
    else
        echo "parameter is empty"
        return $FAILURE
    fi
}

#...
FILE_EXT=${FILE#*.}
echo $FILE_EXT

isNumeric "${FILE_EXT}"
if [ "$?" = "0" ]
then
    echo "Numbered file."
fi