R using ifelse to replace is.na with conditional average / median / mode

I have two columns "Sex" and "Age", where "Sex" is 1 for male and 2 for females. Age is a number presumably between 0.5 and 90 with NA's scattered around.

How do run an ifelse formula so I can fill the average age of Males and similarly Females when the age is NA for that corresponding cell? enter image description here

How do I amend my code to help with this? Current code:

dataset$Age = ifelse(is.na(dataset$Age), ave(dataset$Age, FUN = function(x) mean(x, na.rm = TRUE)), dataset$Age)

I would like have a separate average for Males and Females input in the NA spaces.

Thank you for your kind help in advance.