Adding time differences to a series of files

I am trying to create a function where I batch process a folder of CSV files. All of the CSV files contain timestamps that are incorrect so I have another file with the difference between the timestamp that is incorrect and the corresponding correct timestamp. For example, my files look like this:

df1
ID        Visit    Difference (in seconds)
1002      V2       35
2038      V1       86786

df2
ID        Visit    startTime
1002      V2       2017-12-01 19:47:11
1002      V2       2017-12-01 19:49:55
1002      V2       2017-12-01 19:50:42
1002      V2       2017-12-01 20:18:24

...

I tried to create an if statement to add the difference if the ID and visit number matched

if (df1$ID == df2$ID &
      df1$Visit == df2$Visit) {
    df2$startTime <- df2$startTime + df1$Difference
  }

It repeats adding the 35 seconds then 86786 seconds then 35 etc. so I get an output like this

ID        Visit    startTime
1002      V2       2017-12-01 19:47:46
1002      V2       2017-12-02 19:56:21
1002      V2       2017-12-01 19:51:17
1002      V2       2017-12-02 20:24:50

I want it to only add the 35 seconds. Is there a way to do this?