Suspension of Disbelief
First off, it sounds like this episode broke your suspension of disbelief. In this case, characters behaved in a manner so contrary to how you view them, you couldn't help but view it as some kind of author error. Frankly, there's no getting your suspension of disbelief back, as it's something that has to be maintained during viewing of a work. As I have to answer objectively based on how the characters did behave and not on how you expected them to behave, I can only answer this question by being an apologist for the episode, using the best reasoning I can put forth. However, know that I very much sympathize, as I too have encountered such situations as my suspension of disbelief has been shattered.
Answer
The outcome in a match between Kaguya and Shirogane is not as obvious as you'd think, particularly to these spectators. It's true that Kaguya has built up a good deal of strength from archery and martial art, but Shirogane is by no means unfit, as you've shown. He rides his bike to school each day, and he can be good at sports, if he puts in the Herculean effort necessary. Arm wrestling also appears to not be a competition where his lack of coordination comes into play.
Add into this that human males are naturally stronger than females, and the spectators would have trouble guessing who'd win. A belief in this is displayed by Fujiwara, Ishigami,and Shirogane in this episode. First when Fujiwara asks Ishigami what kind of man he is when he's struggling to carry a bunch of stuff, and second when Ishigami and Shirogane separately state they wouldn't lose to girls in bouts of strength. Fujiwara's belief that males are stronger is only shown implicitly, but Ishigami and Shirogane's proclamations are fairly explicit. We can infer then, that these three will use this belief when calculating who will win a match. Fujiwara will, of course, also take other things into consideration, while Shirogane and Ishigami probably won't since they're fairly confident in this belief.
Next to consider, is when Kaguya and Shirogane are battling it out, they appear to be evenly matched. Of course, we know they were both simply stalling so they could hold each other's hands longer, but the spectators don't know this and are awed by their apparently epic match. Thus, whatever their preconceived notions, Ishigami and Fujiwara truly do believe Kaguya and Shirogane are evenly matched here, because that's what they're seeing.
With all that being said, Kaguya easily defeats Shirogane when she panics about her hands being sweaty. Whether this is from a burst of adrenaline helping her or because she truly did have vastly superior strength all along is unclear.
Also consider that arm wrestling is not entirely about strength. The show even acknowledges this by how Fujiwara tries to cheat by changing her hand position when battling Shirogane. Strategy can come into play, which is something Shirogane and Kaguya are both good at, and thus the outcome is not obvious. See for example: https://www.coachmag.co.uk/exercises/sport-workouts/1552/win-armwrestling
Addendum
To address what you said in the comments about whether Kaguya should even be included:
This battle is very much framed as boys versus girls, thus Fujiwara would have no problem whatsoever including Kaguya despite her apparent edge, because that's advantageous to her side. And Shirogane and Ishigami have overestimated themselves because their opponents are girls, so they wouldn't object either. In fact, it would be rather shameful for them to object. Due to their masculine pride (that might be a strong phrase for these two, but you get the point), they could never say Kaguya is stronger than them and thus shouldn't be allowed to participate.
Manga Version
The manga version does not offer much differences in terms of what happens, but it does offer something helpful at the end: a list of the character's power levels in MP (muscle points).
Men's Average: 60 MP
Women's Average : 40 MP
Shirogane : 64 MP
Kaguya: 62 MP
Ishigami: 55 MP
Fujiwara: 54 MP
Iino: 21 MP
As we can see, the manga actually states that Shirogane is stronger than Kaguya. Thus, while Kaguya did overcome the average male strength level, the work Shirogane put in and his natural strength put him over her by two MP. This means their match actually would be pretty evenly matched.
Some other parts in the manga also support my points.
- Adrenaline may have been involved. While Shirogane and Kaguya are struggling, Fujiwara states, "They're oversecreting adrenaline... their muscle powers have surpassed human limitations!" Note that this does not prove Kaguya won because she got a burst of adrenaline, because that's not what Fujiwara is referring to here. I only point it out because it shows the author was thinking about adrenaline, so it could be the case then. Kaguya's final push does get an additional detail in the manga though: A note stating "A surprise attack!" is there. This might suggest the reason Shirogane lost was because he wasn't expecting Kaguya to get serious so suddenly.
- The spectators truly believe the two are evenly matched. In the manga, it seems to me their reactions to the incredible bout are even more exaggerated. In addition to her above comment, Fujiwara also states all of this:
This is bad... Look at their cheeks! They're so flushed! They're oversecreting adrenaline... their muscle powers have surpassed human limitations! If they continue to wrestle, they might seriously do permanent damage to their arms! Their careers will be cut short... the longer they arm wrestle!
Miko Iino believes Fujiwara, asking the competitors,
There's so much work left to do for the culture festival! Why are you competing so hard?!
Even Ishigami says,
Neither of them is budging an inch. This is a battle of pride!
None of them seems to suspect anything is up with Kaguya and Shirogane, and they don't get anywhere close to suspecting the two are deliberately stalling. Instead, they're all taking this battle with deadly seriousness, to comical effect.
