R - Box Plot Conditional Colors not Working

I'm trying to conditionally color code my box plot, but it's not consistently working. I'm hoping somebody can identify what the problem is. I'm hoping to not use ggplot if I can help it, as I'm pretty new to R and want to stick to what I know (for now).

Essentially, I'm making a series of box plots where 2 of 9 boxes need to be different colors. I can't specify a color pattern because the position of the two boxes change on the x axis for every graph. I have a column labeled "Control" with values of either 0, 2, or 4. I want everything with the value Control=0 to be gray80, Control=4 to be gray40, and Control=2 to be white. I tried to accomplish this in two ways:

#BoxPlot with Conditional Coloring, ifelse statement
boxplot(Y~X, ylab="y", 
 xlab="x", 
 col=ifelse(Control>=3, "gray40", ifelse(Control<=1, "gray80","white")))

#Colors
colors <- rep("white", length(Control))
colors[Control=4] <- "gray40"
colors[Control=0] <- "gray80"

#BoxPlot with Conditional Coloring, "Colors"
boxplot(Y~X, ylab="y", 
 xlab="x", 
 col=colors)

In the boxplot attached, only the first two boxes should be colored in. Can anybody tell me what I'm doing wrong? 1