jQuery - how to test if link goes to anchor on same page?

I'm trying to run an ajax function when links are clicked, but I need to exclude links to anchors on the same page so I don't try to re-load in the page content when I'm simply scrolling down to a different part of the same page.

I know I can test if the href include a hash but that's not good enough:

if (href.indexOf("#") === -1)

Because I will have links that go to another page AND scroll to a local anchor. So I need to test if the href points to the current page AND includes a hash. And in that case I would exclude it from the function. But if it points to a different page and includes a hash it should still be included.

How can I achieve this with jQuery?