What is an alternative to ifelse() in R?

I have a variable (say, VarX) with values 1:4 in a dataset with approximately 2000 rows. There are other variables in the dataset too. I would like to create a new variable (NewVar) so that if the value VarX is 1, the value of NewVar is 0.32 (the value from myMat[1, 1]), if the value VarX is 2, the value of NewVar is 0.05 (the value from myMat[2, 1]) and so on...

myMat
      VarA VarB VarC
[1,] 0.32  0.34 0.27
[2,] 0.05  0.02 0.11
[3,] 0.11  0.11 0.17
[4,] 0.52  0.52 0.45

I have tried the following and it works fine:

df$NewVar <-    ifelse(df$VarX == 1, 0.32, 
                       ifelse(df$VarX == 2, 0.05,
                              ifelse(df$VarX == 3, 0.11,
                                     ifelse(df$VarX == 4, 0.52, 0))))

However, I have another variable (say, VarY) which has 182 values and another matrix with 182 different values. So, using ifelse() would be quite tedious. Is there another way to perform the task in R? Thank you!