jQuery hide all elements without class

I have a simple bit of jQuery which toggles the class "active" to an li once clicked:

$('li').click(function() {
    $(this).toggleClass('active');
});

What I then want to happen is have every other li in the list hidden (display: none) then when you click the active li again it removes the active class but sets all other li's to be visible again too. This is what I am struggling to achieve.

I've tried using an if statement within the click function to check if the "active" class exists and if it does setting all li's to hidden, and if it doesn't then setting the css to show them again, but this doesn't work.