This is only a partial answer, which I'm posting since it seems too long for a comment: I don't know Japanese well and I haven't followed Boruto at all so don't know about the language used in-universe. However, I have some knowledge of kanji from knowing a modicum of Chinese.
To me, some of this writing looks close enough to recognisable kanji, albeit kanji written quickly and sloppily. (Here are some examples of such handwriting, both in Japanese and Chinese: 1, 2. A web search for messy chinese writing generates some fairly good examples in Chinese. I had less luck trying to find explicitly Japanese examples from a quick search in English. Many of the results that appeared were from foreign language learners, whose bad handwriting probably resulted from lack of familiarity with kanji as a whole.) For instance, I can see something that looks like 解, which would be referring to the solution for the problem, on the left side. I can also see 甲 come up several times, but I don't know what it would be referring to. Some of the writing also appears to be Latin/Greek characters used in maths: Vx written in a cursive style is on one of the top lines of the solution.
I don't know if some of the writing here was drawn as meaningless scribbles, but either way, Japanese maths doesn't use a vastly different set of characters from other fields or languages. [Consider that web searches for terms such as 東大 数学 (tokyo university mathematics) will only generate results that look like a combination of normal maths writing and some Japanese characters.]
