My batch script doesn't use if command correctly

I've spent a few days trying to get this batch script to work, but it just does not seem to work properly. It seems to just do whatever it wants after it prompts me to set a variable and i set it.

For example, I might enter n when it says that it doesn't seem to exist, and it will just end the script like it should. But if I re-open it, and it says the same thing as before, and I enter n again, it might just jump to :DeleteCalc, as if I typed y.

Here's my script:

@echo off
:Begin
color fc
title My script
cls
if not exist "C:\calc.exe" (
    echo calc.exe doesn't seem to exist. Attempt deletion anyway? ^(Y/N^)
    set "calcnotexist="
    set /p "calcnotexist="

    ::This command checks to see if the user inputs a quotation mark. If they do, it echos that quotes cannot be inputted.
    setlocal EnableDelayedExpansion
    if not !calcnotexist!==!calcnotexist:^"=! set "calcnotexist="
    endlocal & if "%calcnotexist%"=="" (
        echo ERROR - Quotes cannot be entered.
        pause
        goto Begin
    )

    if /i "%calcnotexist%"=="Y" (
        echo.
        goto DeleteCalc
    )
    if /i "%calcnotexist%"=="Yes" (
        echo.
        goto DeleteCalc
    )
    if /i "%calcnotexist%"=="N" goto End
    if /i "%calcnotexist%"=="No" goto End
    echo ERROR - Unrecognized input
    pause
    goto Begin
)

:calcDoesExist
title My script
cls
echo calc.exe found. Delete? ^(Y/N^)
set "calcexist="
set /p "calcexist="

::This command checks to see if the user inputs a quotation mark. If they do, it echos that quotes cannot be inputted.
setlocal enabledelayedexpansion
if not !calcexist!==!calcexist:^"=! set "calcexist="
endlocal & if "%calcexist%"=="" (
    echo ERROR - Quotes cannot be entered.
    pause
    goto calcDoesExist
)

if /i "%calcexist%"=="Y" goto DeleteCalc
if /i "%calcexist%"=="Yes" goto DeleteCalc
if /i "%calcexist%"=="N" goto End
if /i "%calcexist%"=="No" goto End
echo ERROR - Unrecognized input
pause
goto calcDoesExist

:DeleteCalc
cls
echo Deleting...
if not exist C:\calc.exe goto Success
del /f /q C:\calc.exe >nul 2>nul
if not exist C:\calc.exe goto Success
echo Fail!
echo.
echo calc.exe could not be deleted.
echo.
pause
goto End

:Success
echo Deleted!
echo.
echo calc.exe successfully deleted.
echo.
pause
goto End

:End
exit /b

What could I possibly be doing wrong?

Thanks

P.S. I tested this by opening CMD and running the batch script multiple times in there. (but it also doesn't work right when just double clicking it)