"if" block in VHDL gives bad results

I'm not very familiar with the VHDL language but I managed to write this code for a 4 bit substractor using two's complement:

signal result: std_logic_vector(4 downto 0);
signal As: std_logic_vector(4 downto 0); -- A with the sign bit
signal Bs: std_logic_vector(4 downto 0); -- B's complement with the sign bit
signal comp : std_logic_vector (3 downto 0); -- B two's complement
signal s : std_logic; -- sign bit

begin                     
    As<='0'&A;
    comp<=(not B)+('0'&'0'&'0'&'1');
    Bs<='1'&comp;
    result <= As+Bs;
    s <= result(4);
    process (s)
        begin
    if (s = '0') then
        diff <= result(3 downto 0);
    else
        diff <= (not result(3 downto 0))+('0'&'0'&'0'&'1');
    end if;
    end process;

the problem is that I get wrong results in "diff" and I can't locate the issue however when I remove the "if" block I get the right result but only when the difference is a positive number, so I'm using "if" to manage both cases