You made a mistake in thinking his soul could come back as many times as it pleases. A Yomi Yomi no Mi user can only be resurrected once. He does seem immortal as we have seen him be beheaded and damaged several times, from which he had healed back by drinking calcium rich drinks such as milk. The only two possibilities there are to killing him is by either submerging him into water or by crushing his entire body forcing the soul to leave the body. This would indeed make him quite immortal.
How many times does the Yomi Yomi no Mi resurrect the soul?
The wiki page on the Yomi Yomi no Mi states:
The major strength of the fruit, as demonstrated by Brook, is that it gives its user another life after dying once before.
The wiki page on Brook states:
While he was still alive, Brook ate the Yomi Yomi no Mi, a Paramecia-class Devil Fruit. Should the user of this fruit die, it allows them the one-time opportunity to return to life.
Will the soul be able to come back regardless of the state of the user's body?
The wiki page on the Yomi Yomi no Mi states:
After death, the user continues to exist in whatever state they were revived as and can even go as far as to grant a form of "immortality" that lasts until the user suffers a second death. This of course is negated if the user's cause of death was drowning, and their body remains in the water, or if their body is completely destroyed.
What will kill a Yomi Yomi no Mi Devil Fruit user?
The wiki page on Brook states:
One advantage is that because he is just comprised of light weighted bones, he can use his speed and light weight to run across water, but, like all Devil Fruit users, he can still drown when submerged.
The wiki page on the Yomi Yomi no Mi states:
After death, the user continues to exist in whatever state they were revived as and can even go as far as to grant a form of "immortality" that lasts until the user suffers a second death. This of course is negated if the user's cause of death was drowning, and their body remains in the water, or if their body is completely destroyed.
What will not kill a Yomi Yomi no Mi Devil Fruit user?
The wiki page on Brook states:
He has also proven that calcium (such as in milk) will rapidly heal any damaged bones.
If the user's bones are only partially destroyed, he will be able to regenerate them through consumption of calcium. Eg This was seen after Brook fought Ryuma and was severely damaged.
The wiki page on the Yomi Yomi no Mi states:
As long as his bones are not hurt any attacks will not go to his core like when Zeo beheaded Brook, his soul managed to repair it and connect his bones together again.