If I'm not mistaken he's not quite talking about "enjoying" the food as much as he is talking about "understanding" the food. I say this because of two pages in particular. (I believe one is the one that you quoted from.)
Firstly, in Chapter 133 page 12 he states that "True Gourmet Food, It greatly resembles excellent paintings, sculptures, and music." Then he goes on to state "The true worth of all first rate art can only be understood by those who have a certain quality of character." I believe he's implying that most of the people in the room (possibly the world also, he is pretty arrogant) are just amateur spectators and don't really know, and will never know, what "True Gourmet Food" is.
Secondly, in Chapter 133 on page 13 he states that the only thing that should be called "Cuisine"/"Gourmet Food" Shoul be the food for the select few that understand it's worth (The Above Paragraph). Any other food is just fodder. Which further backs my point that I made that says he thinks that majority of people won't know what "Gourmet Food" is. I say this because Fodder is a food for livestock and I'm assuming based on his personality that he is comparing the general public to cattle.
It might be that he is talking about a taboo/forbidden style of gourmet, but I think that it is more likely that he is just talking himself up and talking others down.