C Program Won't End When User Enters Invalid Character

I'm trying to write a C program that computes X^Y (X to the Y power) without using the 'pow' function. The program works fine when I enter numbers, but I'm having an issue with the code not stopping when the user enters a character that isn't a number. It runs through the program once more after giving the message "The character you have entered is not a number.Please try again.The character you have entered is not a number.Please try again. The value is 1." Can someone help? I'm going insane trying to figure this out.

#include <stdio.h>
#include <stdlib.h>

int main()
{
int x, y, i,n=1,val; 

printf("Please enter a number: \n"); 
scanf("%d", &x);

if (x!=1)
    printf("The character you have entered is not a number.Please try again.");

else
    printf("What power is the number raised to?\n");
    scanf("%d", &y);

    if (y!=1)
    printf("The character you have entered is not a number.Please try again.");

    else
        for(i=1;i<=y;i++)
        {
        val=x;
        n=n*val;
        }
    printf("The value is %d.\n", n);
return 0;
}