How to exit a while loop that uses numbers with a character that doesn't mess up the math

I have a script to write that is able to take user input of however many numbers they want until they press "q". Now I have it working, but it takes q and uses it in the math that the other numbers should be being used in. It also uses it to show what is the highest or lowest number.

total=0
count=0
largest=num
smallest=num
while [ "$num" != "q"
do
      echo "Enter your numbers when you are done enter q"
      read num
      total=`expr $total + $sum`
      count=`expr $count + 1`
      if [ "$num" > "$largest" ]
            then
                   largest=$num
      fi
      if [ "$num" < "$smallest" ]
      then
                   smallest=$num
      fi
done
avg=`expr $total / $count`
echo "The largest number is: $largest"
echo "The smallest number is: $smallest"
echo "The sum of the numbers is: $total"
echo "The average of the numbers is: $avg"