Python higher lower number guessing game

In this program, you think of the number, the computer guesses. Before the game begins, the computer asks how many guesses it gets. If the computer loses, it asks what the correct answer was. It also checks to make sure that the answer is legal, and points it out if it wasn’t. If the user gives inconsistent answers, the computer points it out and stop playing.

My issue is that when I run the program, if I say "higher" or "lower" when the computer asks if my number is higher/lower/same as a number, say 50, I can later say my number was 50 and it tells me that I win instead of saying "That can't be; you said it was higher/lower than 50!" How would I fix this problem?

print("Think of a number between 1 and 100 and I'll guess it.")
total = int(input("How many guesses do I get? "))

h = "higher"
l = "lower"
s = "same"
low = 0
high = 100
hls = ""

guess_count = 0

average = 0

while guess_count < total and hls != s:
    average = (high + low) // 2
    hls = input("Is the number higher, lower, or the same as " + str(average) + "? ")
    if hls == h:
        low = average
    elif hls == l:
        high = average
    guess_count += 1
    if high - low == 1:
        break
if high - low == 1:
    print("Wait; how can it be both higher than " + str(low) + " and lower than " + str(high) + "?")
elif hls == s:
    print("I won!")
else:
    answer = int(input("I lost; what was the answer? "))
    if answer < low:
        print("That can't be; you said it was higher than " + str(low) + "!")
    elif answer > high:
        print("That can't be; you said it was lower than " + str(high) + "!")
    elif answer != average:
        print("Well played!")