Batch: Why does the parsing behavior of a variable of a FOR change within an IF?

Why is this batch not working:

IF "%videoFile%"=="" (
    SET videoExt=mp4
    FOR /F "delims=*" %%a IN ('dir /b *.%videoExt%') DO (SET videoFile=%%~nxa)
)

While this works:

IF "%videoFile%"=="" (
    FOR /F "delims=*" %%a IN ('dir /b *.mp4') DO (SET videoFile=%%~nxa)
)

And as a smart guess this works too:

IF "%videoFile%"=="" (
    SET videoExt=mp4
    FOR /F "delims=*" %%a IN ('dir /b *.%%videoExt%%') DO (SET videoFile=%%~nxa)
)

Is there maybe another way possible?