I’m guessing you have items sorted by rating; I’d change that to something else.
In case it helps, I rate items by using the fullscreen view and using the keyboard shortcuts. My default sort order is by filename. I don’t ever use the sort-by-rating option. I do, however, often use a filter to display only images above a certain rating.