Excel IF Function reading non-numeric value as greater than a numeric value

I am attempting to use the formula below to calculate residuals from the slope (X1) and intercept (X2) of a regression. However, I am running into the issue that when there is a non-numeric value in X6 (e.g. N/A) the IF function is returning a '1'. In other words, N/A in being interpreted as greater than a numeric value, which is misleading...

=IF(X6>=((X1*E6)+X2),"1","0")

What am I missing? I would expect a non-numeric value to return an error when using >=.

I have attempted to nest IF functions to get around this [variations of =(IF(X6=N/A, "N/A",IF(X6>=((X1*E6)+X2),"1","0")] but couldn't get it to work.

Thanks for any advice!