ifelse statement returning a conditional matrix element in R

I'm trying to test each line of my data frame for a condition without using a forloop in r, and have the return be an element of a separate matrix that corresponds to the conditions. Here is an example:

ax <- matrix(data=c("x","UP","DN","x"),nrow=2,dimnames=list(c("site1","site2"),c("site1","site2")))
data <- data.frame(Location1=c("site1","site1","site2","site1","site2","site2","site2","site1"),Location2=c("site1","site2","site1","site2","site2","site2","site1","site1"))

This results in a matrix (ax) and data frame (data):

   > ax
        site1 site2 
  site1 "x"   "DN"  
  site2 "UP"  "x"


> data
  Location1 Location2 
1    site1     site1        
2    site1     site2        
3    site2     site1       
4    site1     site2        
5    site2     site2        
6    site2     site2        
7    site2     site1       
8    site1     site1        

Now if location1 is different than location 2, I'd like to get the corresponding matrix element that tells me the direction of that movement. I've used ifelse statements like this before, but getting the correct output is really eluding me when trying to query a separate matrix.. My code is:

data$movement <-ifelse(data$Location1!=data$Location2,ax[as.character(data$Location1),as.character(data$Location2)],"x")

But this results in this output:

data
  Location1 Location2 movement
1    site1     site1        x
2    site1     site2        x
3    site2     site1       UP
4    site1     site2        x
5    site2     site2        x
6    site2     site2        x
7    site2     site1       UP
8    site1     site1        x

It seems like this would be an easy problem, but I can't seem to figure it out, any help is greatly appreciated