How can I prevent IF statement from running once the condition is met

I'm making a game where players have to input numbers to guess the population of a country, but I'm having issues counting their scores. I have this if statement:

 if (player1 === "" || player2 === "") {
          resultSpan.innerHTML = "One or two empty values";
        } else {
          resultSpan.innerHTML = `The correct number is ${population.toLocaleString()}`;
          let playerWin = document.querySelector(".results-player-win");

          if (
            Math.abs(parseInt(population) - playerOneTotal) <
            Math.abs(parseInt(population) - playerTwoTotal)
          ) {
            playerWin.innerText = "Player 1 wins!";
            playerOneScore++;
          }

          if (
            Math.abs(parseInt(population) - playerOneTotal) >
            Math.abs(parseInt(population) - playerTwoTotal)
          ) {
            playerWin.innerText = "Player 2 wins!";
            playerTwoScore++;
          }
          document.getElementById(
            "current-score"
          ).innerHTML = `P1: ${playerOneScore} --- P2: ${playerTwoScore}`;
        }

But, instead of incrementing the score by 1, sometimes it's two or three. I'm guessing this is because another eventListener is being called beforehand. However, I can't find a way to separate them completely, as they both need to use the same population variable for the score to be calculated correctly.

I've tried adding this.remove() inside of the IF statements, but it deletes the button. I've also tried e.stopImmediatePropagation(); and creating a variable isFirst, and setting to true then false, but they block the code completely on second execution.

For more clarity, I'm also posting a JSFiddle.