Koyomi's thought processes are elucidated upon in the light novel.1
The events of this scene are ordered slightly differently in the light novel versus the anime. What happens in the light novel is that immediately after Nadeko hands Kanbaru's swimsuit to Koyomi (08:09 on the BD), their conversation comes to a halt. Then, the events you're talking about occur. It would probably be more instructive to just see the ensuing text rather than have me describe it.2 Keep in mind that Koyomi is the narrator here. Editorial inclusions in square brackets.
I reached my hand out to touch Sengoku's bangs.
"...oh?" [Koyomi]
I missed.
My hand swept through the air. Sengoku quickly moved her head to the side, dodging my hand. So then, you know, I reached for her bangs again, but she took a step away from me, thus evading my follow-on.
"W...what?" [Nadeko]
"Uh, well..." [Koyomi]
Does it really bother you that much?
Her movements were surprisingly quick - nothing like what you'd expect from a quiet girl like her. You'd think that the bangs would interfere with her vision when they fell in front of her eyes, but she didn't seem to care.
"Hm." [Koyomi]
Well, let's try this.
I quickly extended my other hand and lightly lifted the hem of her dress. Given the way she reacted to me trying to touch her bangs, I was expecting her to respond like a little girl whose skirt had been flipped. Hence that little experiment.
But Sengoku didn't react to that at all. She looked at me a little curiously and tilted her head to one side.
It struck me yesterday, too, but....
She's way too innocent for a middle schooler.
She's worried about all the wrong things.3
My best interpretation of what's going on here is that Koyomi is just messing with Nadeko, trying to get a rise out of her. At first, he goes for her bangs, and Nadeko recoils. Koyomi decides to take it one step further and lifts the hem of her dress, expecting a bigger reaction out of her - but she doesn't seem to care. From Koyomi's point of view, this is all backwards - why would she be more bothered by someone touching her hair than by someone lifting her skirt? Hence his comment at the end of this bit - she's worried about all the wrong things.
It will probably be helpful to also look at this from Nadeko's point of view, particularly in light of Otorimonogatari. As we learn, Nadeko really hates it when people touch her bangs - she's very self-conscious about her hair, hence that hat she often wears. The skirt-lifting, though, probably doesn't bother her, seeing as the person doing the lifting is Koyomi - and by this point, she's long since fallen in love with Koyomi.
1 Bakemonogatari chapter 5 [book 2], section 002. (i.e. 化物語（下） 第五話（つばさキャット） ００２)
2 Translation my own; any falsities in it are my fault.
3 This is, I think, a better rendering of the line that appears on the sign in your last image.
