Seki-kun always plays in class or even outside the class (PE or even during disaster drills). Does he ever pay attention at all? What about tests or his homework? Has he ever been caught playing in class? Usually Rumi is the one who gets scolded by the teacher because she was paying attention to Seki.
Maybe it is not shown in anime or manga directly, but maybe there is something along the line of "He only pays attention to some teacher" or "He used to always pay attention before".