looping two lists and trying to find what doesn't match for a certain field brings wrong results in python

I have two lists:

file_a = [['id', 'source_1', 'source_2'], ['1', 'source_a1', 'source_a2']]  

file_b = [['id', 'source'], ['3', 'source_b']]

Now I'm iterating through the lists to see what matches and what doesn't.

match = []  
missmatch = []  

for row_a in file_a:
    for row_b in file_b:
        if row_a[1] == row_b[1] or row_a[1] == row_b[2]:
            match.append(row_b])
        elif row_a[1] != row_b[1] or row_a[1] != row_b[2]:
            missmatch.append(row_b)  

The match is working, but not the missmatch. I know why it is wrong, but I can't find a solution to get it solved.

Any help appreciated.

Thanks Manuel