using a DelayedExpansion index variable for an array within an IF statement fails

I have a batch file where I am displaying all *.pem files in a directory, giving the user an option to choose one, and at this time just trying to show the user what file they picked by using the number they chose to ECHO the contents of the array. I believe that somehow because it is inside of the IF statement, the index variable "selectedPem" is not expanding in the array's index brackets because it is using the % and not using the delayed expansion !, which doesnt appear to work in the array's index brackets. I think this because if I set the "selectedPem" variable before the IF statement, it ECHOs properly. I think the only option for this to work is to somehow make use of a subroutine. Why is selectedPem variable not working inside of the IF statement?

@echo off
setlocal enabledelayedexpansion 
set dir=%~dp0
cd %dir%

ECHO Performing initial search of private and public keys...
set pemI=0
set keyI=0
for %%p in (*.pem) do (
    REM echo %%~nxp
    set pems[%pemI%]=%%~nxp
    REM echo !pems[%pemI%]!
    set /a pemI=%pemI%+1
)
REM echo %pems[0]%
set /a totalPems=%pemI%-1
REM This is the test selectedPem variable that showed me the variable works 
REM if I set it outside 
REM of the "if defined pems[0]" statement
REM set selectedPem=
if defined pems[0] (
    echo PEM Files Found:
    for /l %%p in (0,1,%totalPems%) do (
        echo %%p^) !pems[%%p]!
    )
    ECHO Select a pem file to be used as your private key. Otherwise, press 
    Q to not select one.
    set /P selectedPem=Enter a number or Q:
    IF "!selectedPem!"=="q" (
        ECHO Skipping private key selection.
    ) ELSE (
        ECHO !selectedPem!
        ECHO %pems[0]%
        REM The below ECHO only works when the above selectedPem is set 
        REM outside of the "IF defined" statement
        ECHO !pems[%selectedPem%]!

        REM Tried these other implementations:
        REM ECHO !pems[!!selectedPem!!]!
        REM ECHO %pems[!selectedPem!]%
        REM setlocal disabledelayedexpansion
        REM ECHO %pems[%%selectedPem%%]%

        set privatekey=!pems[%selectedPem%]!
        ECHO You chose %privatekey%
    )   
)`

Output:

Performing initial search of private and public keys...
PEM Files Found:
0) brandon.pem
Select a pem file to be used as your private key. Otherwise, press Q to not 
select one.
Enter a number or Q:0
0
brandon.pem
ECHO is off.
You chose

I understand that "ECHO is off." is outputted instead because it is interpreting my array variable reference as empty. I appreciate your time in reading over my script, I am likely over-complicating it.