If statement never evaluates false when it should

Another amateur issue with a script I am working on. I am having an issue with exiting my script and console from within an If statement.

I have tried exit, exit(), $host.exit and [Environment]::Exit(1) within my Else portion. No matter what the user input is, the if statement continues.

I feel like I am missing something very simple but I can't figure it out. I only want it to continue if the user enters "Y" or "y". Anything else I just want it to fail and close...

I have also called $confirm after the user input make sure it's capturing the correct letter, which is is.

Here is the portion of the script that is failing:

$confirm = Read-Host "Continue deletion? [Y/N]"

if ($confirm -eq "Y" -or "y") {
    for ($i = 0; $i -le 4; $i++) {
        Get-ChildItem -Path $path0, $path1 -Filter "$dval*"| 
        % { Write-Host 'Deleting' $_.FullName $dval; Remove-Item $_.FullName -Recurse }
        $dval++ } 

} else { [Environment]::Exit(1) }