Use ifelse to assign Date in new factor

I would like to assign different dates to new factor in a dataframe that is dependent upon the year of the particular observation. Here's what I have:

Date     Location
2014-10-30     AB1
2014-09-02     AB3
2015-10-15     AB1
2016-11-02     AB4
etc. (191K observations)

And here's what I'd like to get, given I have Reference dates of 2014-08-14, 2016-09-16, etc. associated with the Year of the observation's date. For those years without a Reference date, I'd like the value set to NA.

Date     Location     Reference
2014-10-30     AB1     2014-08-14
2014-09-02     AB3     2014-08-14
2015-10-15     AB1     NA
2016-11-02     AB4     2016-09-16
etc. (191K observations)

I know how to write the ifelse statement to specify the assignment of dates to particular years, but the syntax of identifying a particular year based on 'Date' and matching the Reference date based on that value isn't obvious to me. I would really appreciate some guidance on this issue. Thanks. Doug