Well, somebody has to do the grunting.
It may sound like I'm joking, but I'm really not. I don't know if canned grunts or grunt SFX packages exist, but I don't think the producers would want to use them - Seki has to grunt in a number of different ways and in a number of different contexts, and when you get down to it, Seki's grunts can really be quite emotive.
You could, in principle, have Seki be completely soundless, but that would be unrealistic - unlike the dude from Non Non Biyori, he does interact with the other characters to a substantive degree, and if he didn't make a single noise, it would be kind of weird.
For illustrative purposes, I have compiled a collection of Seki grunts and related noises from episodes 1 and 2 (http://vocaroo.com/i/s0PuflktAAbw; do forgive the shoddy editing). I hope you will agree that this stunning performance on Shimono Hiro's part is reason enough to have a voice actor for Seki. For an even better illustration, watch episode 16 from roughly 4:20 until the ending credits (the part where Yokoi is messing with him) - he vocalizes a lot there.