ggplot rescaling legends via NA (then backfilling NA values)

library(tidyverse)
heatmap <- tibble(col1 = c(rep("A", 3), rep("B", 3), rep("C", 3)),
                  col2 = rep(c("I", "II", "III"), 3),
                  col3 = c(1:8, 99))
#> # A tibble: 9 x 3
#>   col1  col2   col3
#>   <chr> <chr> <dbl>
#> 1 A     I         1
#> 2 A     II        2
#> 3 A     III       3
#> 4 B     I         4
#> 5 B     II        5
#> 6 B     III       6
#> 7 C     I         7
#> 8 C     II        8
#> 9 C     III      99

I can create a heatmap from the data frame above that looks like this.

ggplot(heatmap, aes(col1, col2)) + geom_tile(aes(fill = col3))

The C/III value is 99 and makes the scale of the legend almost unusable. With that in mind I wrote the following code to rescale the legend for a "better" view on things.

ggplot(heatmap %>% mutate(col3 = ifelse(col3 > 10, NA, col3)), 
       aes(col1, col2)) + 
  geom_tile(aes(fill = col3))

I want the end user to know the value of the original "NA" squares, but I don't want to clutter the rest of the heat map with labels for every other square. What I want is shown below.

How do I backfill my NA values to be able to use them with geom_text(), and how do I limit ggplot to only drawing labels on top of grey NA squares, not "normal" squares. The beginning of this may look something like the following.

ggplot(heatmap %>% mutate(col3 = ifelse(col3 > 10, NA, col3)), 
       aes(col1, col2)) + 
  geom_tile(aes(fill = col3)) + 
  geom_text(aes(label = col3))