Using ifelse to determine point size in r ggplot, works until legend is added

I am trying to set up a graph where the size of the point is smaller if n == 0 than n > 0. The code works until I add a legend. Here is my code that works:

ggplot(len.oo, aes(x = TCL, y = n, colour = worm, shape = worm)) + ylim(0, 20) + geom_point(size = ifelse(len.oo$n == 0, 2, 4)) + theme_bw() + xlab(expression(~italic("O. obscurus")~"TCL (mm)")) + ylab("Abundance") + theme(legend.title=element_blank(), legend.position="none")

that gives me (apparently i cant post images):

ggplot without legend

as soon as i add a legend it gives me an error. code with legend: ggplot(len.oo, aes(x = TCL, y = n, colour = worm, shape = worm)) + ylim(0, 20) + geom_point(size = ifelse(len.oo$n == 0, 2, 4)) + theme_bw() + xlab(expression(~italic("O. obscurus")~"TCL (mm)")) + ylab("Abundance") + theme(legend.title=element_blank(), legend.position=c(0.2, 0.8)) + guides(size=FALSE)

gives me the error: Error: Aesthetics must be either length 1 or the same as the data (3): size

I've also tried position = "top" and the like to have the legend outside the plot and without the guides(size=FALSE)

I could easily make the graph without the legend and then make the legend without the size difference and use other software to copy and paste the legend on the image, but I would like to do all this in R.

thank you