How to merge two datasets using if-function in R when datasets are of variable length

I have GPS points (lat & long) recorded every hour for 60 days, and I have sunrise and sunset times for the same time frame. I am trying to merge the two data sets, so I know if the sun was up or down when GPS point x was recorded.

I have managed to make it work if the sun dataframe is smaller than the gps dataframe. But if the sun dataframe is larger (as shown in the example below), I get an error. Unfortunately, I would need the code to work under both circumstances (i.e., regardless which dataframe is larger).

Example data and code:

library(lubridate)

gps <- data.frame(lat = c(54.008, 54.009, 54.009, 54.008, 54.009, 54.009),
                  long = c(38.050, 38.051, 38.053, 38.050, 38.051, 38.053),
                  date = as.Date(c("2019-12-19", "2019-12-19", "2019-12-19", "2019-12-19","2019-12-20", "2019-12-20"), format = "%Y-%m-%d"),
                  time = as.numeric(hm("06:00", "07:00", "12:30", "13:00", "07:00", "24:00")))

sun <- data.frame(date = as.Date(c("2019-12-19", "2019-12-20", "2019-12-21", "2019-12-22", "2019-12-23", "2019-12-24", "2019-12-25"), format = "%Y-%m-%d"),
                  sunrise = as.numeric(hm("06:04", "06:05","06:06", "06:07","06:08", "06:09", "06:09")),
                  sunset = as.numeric(hm("23:06", "23:06","23:06", "23:06","23:06", "23:06", "23:08")))

gps$sunlight <- ifelse(sun$date == gps$date | sun$sunrise >= gps$time | sun$sunset <= gps$time, "N", "Y")

Error:

    Error in `$<-.data.frame`(`*tmp*`, sunlight, value = c("N", "Y", "Y",  : 
  replacement has 7 rows, data has 6

Wanted output:

    > gps
     lat   long       date  time sunlight
1 54.008 38.050 2019-12-19 21600        N
2 54.009 38.051 2019-12-19 25200        Y
3 54.009 38.053 2019-12-19 45000        Y
4 54.008 38.050 2019-12-19 46800        Y
5 54.009 38.051 2019-12-20 25200        Y
6 54.009 38.053 2019-12-20 86400        N

Any suggestions or ideas where I may be going wrong?