In Brave 10, when a character goes to use a spell's incantation or uses a chant for a secret technique, the subtitles don't have any English words in it (I am referring to the home release by NISA, not fansubs). However, it does sound like they are saying something in Japanese. An example of this is in episode 2 when Hanzo Hattori goes to use Pure Flame Samadhi on Saizo.
I am wondering, what language are the spells/chants being said in? Or did they just throw random characters together?
What language is it?
That's Japanese. It's not ordinary contemporary Japanese, though, but rather Buddhist Japanese, which has some unusual qualities that make it rather unlike contemporary spoken Japanese.
Why isn't it translated?
Buddhist texts in Japanese are strange things. As the informed reader is aware, Buddhism originated in India, and as such, many of the foundational texts of Buddhism were originally written in Sanskrit. When Buddhism moved into China, those Sanskrit texts were translated into whatever the contemporary form of Classical Chinese was. Eventually, those texts made their way from China into Japan.
I'm glossing over a lot of details here to get to the important point: most Japanese Buddhist terminology was first introduced around the 6th century CE. Since these were loanwords from Chinese (many of which were in turn loans from Sanskrit), they used Sino-Japanese pronunciations (that is, on readings, and in particular, go-on readings). Buddhist terminology has changed relatively little since.
Modern Japanese consists primarily of 1.) native words (e.g. kun readings); 2.) non-Sinitic loanwords (e.g. borrowings from English); or 3.) Sinitic loanwords that postdate the 6th century (that is, the kan-on and tō-on readings). go-on vocabulary (like Buddhist words) are relatively scarce in the contemporary Japanese language, and the average speaker is unlikely to be particularly familiar with many go-on words.
Making things more difficult still is the fact that Japanese Buddhist texts are often straight-up Chinese texts simply being read in Japanese pronunciations, making it difficult for someone without training in reading Buddhist texts to understand what is being spoken.
So that's probably why it wasn't translated - it was too difficult for whoever was on hand. It's certainly possible to translate whatever was said (see below), but I guess it wasn't worth the trouble for the people at NISA. See also footnote4. Anyway, that's enough of that digression.
What is he actually saying?
If my ears are working today, what Hattori Hanzo says in episode 2 is:
オン ソンバ ニソンバ ウン バザラ ウン パッタ
on sonba nisonba un bazara un patta
Now, what does this mean? Heck if I know - I certainly don't know how to read Japanese Buddhist texts. Luckily, the internet has answers for us. This is apparently the mantra1 (in Japanese, shingon) for a being called 降三世明王 (Gouzanze Myouou; Skt. Trailokyavijaya)2, one of the five Wisdom Kings.
The Sanskrit equivalent of this chant appears to be:3
om sumbha nisumbha hum vajra hum phat4
I'm not going to attempt to translate this, because my Sanskrit is far too weak for that, but some of the relevant things to know are: Sumbha is an alternate name (or epithet or something like that) for Gouzanze Myouou, as is Vajrahumkara, here contracted to Vajra. Nisumbha is the name of the other asura that appears in stories detailing the exploits of Sumbha. "Om" is of course the sacred syllable Om. The rest I'm honestly not sure about, but I'm sure a better-educated person would be able to translate this.
* If you find this sort of thing interesting, take a look at the Buddhism proposal over at Area51! They need more committers!
1 Full text for the mantrams of the other Wisdom Kings here, I think.
2 See also this JAANUS article.
3 According to this guy and also the article on shingon on Japanese Wikipedia.
4 The astute reader will observe the startling similarity of the Sanskrit version of the mantra to the Japanese version, this despite Sanskrit and Japanese having no genetic relationship to one another. This is another reason why Buddhist mantras in particular are difficult to understand - they are often just plain Sanskrit processed through Chinese without translation and then readapted to fit Japanese phonology, and as such are often devoid of any actual Japanese words.
