how to make conditional ggplot geoms

library(tidyverse)

df1 <- data.frame(Name = c("Jack", "Jill", "John"),
                  Qty =  c(3, 8, 4), 
                  Violation = c("F", "T", "F"))

ggplot(df1, aes(Name, Qty, group = 1)) + 
  geom_line() +
  geom_point() + 
  geom_hline(aes(yintercept = 5), linetype = "dashed") + 
  geom_point(data = df1 %>% filter(Violation == "T"), color = "red")

The code above highlights any data point meeting the Violation == "T" criteria. However, my data frames change, and sometimes don't contain any data points meeting Violation == "T". When this happens I get an error, like below:

df2 <- data.frame(Name = c("Jack", "Jill", "John"),
                  Qty =  c(3, 2, 4), 
                  Violation = c("F", "F", "F"))

ggplot(df2, aes(Name, Qty, group = 1)) + 
  geom_line() +
  geom_point() + 
  geom_hline(aes(yintercept = 5), linetype = "dashed") + 
  geom_point(data = df2 %>% filter(Violation == "T"), color = "red")

#> Error: Aesthetics must be either length 1 or the same as the data (1): x, y

Do I need to construct some type of ifelse() statement to make my ggplot code work whether there are any Violation == "T" or not?