User input and stuck in while loop

I want my code to take a user input, then prompt the user to continue, if they answer 'y' then it asks for another input, if they answer 'n' the program stops and if they type any other characters it simply continues to prompt them until they enter a 'y' or 'n'.

As the code shows I'm trying to use a while loop to continuously prompt the user until they enter a 'y' or 'n'. However when I reach the while loop it does not stop when a 'y' or 'n' is entered.

def test():

    number = input('Input a number then press enter:')    
    print(number)
    prompt = input('Continue (y/n)? ')

    if prompt == 'y':
        number = input('Input a number then press enter:')
        print(number)
        prompt = input('Continue (y/n)? ')
    elif prompt == 'n':
        pass

    else:
        while prompt != 'y' or 'n':
        prompt = input('Continue (y/n)? ')