how to make css apply to both logic or one of them in jquery

I have an if statement.

I want in case of condition A do something for input A but not for input B.

vice versa, if condition B do something for B. but not for input A.

but if both conditions occur, do it for both.

but my code, if one of them is empty then both will change the background.

HTML:

<input type="text" class="x" />
<input type="text" class="y" />
<input type="submit" value="sum" class="submit" />

JS:

$('.submit').click(function() {
  if (!$('.x').val() || !$('.y').val()) {
    $('.x, .y').css('background-color', 'blue');
  } else {
//sum
  }
});

JSFiddle