Fill iterates over each pixel of the referenced image(s), so you can imagine the commands of your fill string running each time it moves to a new pixel. A simple example of removing pixels with fill:
gmic sp dog f "x%2?i:0"
which is saying “if the current x value modulo 2 is 1 then keep the current pixel, else replace with 0”.
More complex things can be done by referencing pixels of another image, for example by using i(#index,…).
Vectors at most basic can be good for operating on several values at once, e.g. if you multiply I
you’re multiplying R, G, B each by the same. The usual vector functions are available too. I wouldn’t suggest replacing values with nan because it’s usually more of a problem to handle than just setting some extreme value (very large/very negative/very small), or keeping a mask in another image. Comparison operators on the image as a whole might help (eq/gt/lt…) as well as min/max. Maybe not the best way to learn, but most of what I use was gleaned by reading other filters which did a similar thing.
It’s quite ‘low level’ for this sort of thing and not a lot different from working with buffers in something like C. Loops, logic and maths!