proper usage of 'if not in' python

I'm working through project euler problems, specifically problem 23:

So I figured out a solution in python but it was running very slowly and I realized that I was storing multiples of the abundant sums in a list. I tried to prevent this by adding a 'not in' but for some reason adding this single line changed my answer completely. Any ideas why this is happening?

for i in range(13, 28123):
    s = sum_divisors(i)
    if s > i:
        abundant_numbers.append(i)

check_list = list()

for i in abundant_numbers:
    for j in abundant_numbers:
        s = i + j
        if s <= 28123 and s not in check_list:  # only check when sum < 28123 and prevent duplicates
                check_list.append(s)
        else:
            break

check_list.sort()
sum_ans = 0

for i in range(28123):
    if not binary_search(check_list, i):
        sum_ans += i

print 'sum', sum_ans

When I leave it as just

if s <= 28123

it works fine. The problem only occurs with the 'not in' condition.