How to find max and min numbers in list within list

I am stuck at this question where I am required to update all largest and smallest numbers in the list with the average value as a way to remove the extreme values in the given list.

For example:

def remove_extreme( [ [0,4], [1,4], [-1,2] ] ) would return [ [0,1.5], [1,1.5], [1.5,2] ].

Here's my code:

def remove_extreme(datalist):
    for numlist in datalist:
        for index, number in enumerate(numlist):
            largest = max(number)
            smallest = min(number)
            average = (largest - smallest)/2
            if number == largest or smallest:
                num_list[index] = average
        return datalist

May I know what went wrong? I keep getting 'int' object is not iterable.