My own filter that I had pushed into gmic-community gives me this:

If I can figure out how to extract those pesky out of place pixels as a single channel mask, I believe this can be solved.
Also, my filter works for images with larger blur too.
EDIT: Getting there.

Now, I have to do indexing in-place via math parser. That’s not the fun part.
Current code:
#@cli rep_color_region: _threshold[%]>0
#@cli : For use in images with large color blobs, this simplifies a image.
rep_color_region:
skip ${1=25%},${2=2%}
threshold={cut($1,0,1)}
foreach {
+round
+colormap. 0,,0
+index[-2] [-1],0,0
+area. 0,0
ge. {ia*$threshold}
*.. .
negate.
-.. .
rm.
colormap. 0,,1
crop. 1,100%
map. ..
rm..
index.. .,0,1
rm. rv
100%,100%,100%,1
1,1,1,2
eval[0] >"
begin(
off_place=[-1,1];
);
pixel_in_place=0;
repeat(4,position,
xp=off_place[position&1];
yp=off_place[position>>1];
pixel_in_place=(I==J(xp,0,0,0,1))&&(I==J(0,yp,0,0,1));
if(pixel_in_place,break(););
);
if(!pixel_in_place,
pixel_position=[x,y];
i(#-2,pixel_position)=1;
da_push(#-1,pixel_position);
);
I;
"
area_fg.. 0,0
inrange.. 0,{iM*$2},0,1
eval "
size_da=da_size(#-1);
point=size_da-1;
repeat(size_da,
if(!i(#-2,I[#-1,point]),da_remove(#-1,point););
--point;
);
da_freeze(#-1);
"
}
Last thing to add is a eval which do inplace indexing with surrounding pixel that are not part of the mask as color reference using the 1d strip image.
Also, allow it to utilize 3 dimensions, but I’ll do that much later.
That being said, even when all of this is done, some human intervention would have to be used. Not a big deal though.