That episode/arc was the first to introduce such a Haki limitation concept. In fact, the wiki page on Haki cites that set of episodes to state that:
Despite their great capabilities, Haki is not limitless as it can be depleted from overuse, rendering the user unable to use it for a set period while it regenerates.
Also taking from that page the general definition:
Broadly speaking, there are two types of Haki available to everyone, given the proper training, but there is a third type that only a certain group of "chosen ones" are said to possess. In simple terms, Haki is an ability to sense and predict spiritual energy (Kenbunshoku), use life force as physical reinforcement (Busoshoku), and, for the rare "chosen ones", overpower enemies' willpower with your own (Haoshoku).
Thus it is only Haoshoku or Conqueror's Haki that is directly tied in to their willpower. An example of this would be:
In Chapters 923 and 924 of the Wano arc, we see that Luffy's willpower manages to make some weaker soldiers faint even though he was knocked out by Kaido. Kaido himself says that Luffy has been glaring at him.
As you said, none of the Haki forms are specific to a user's body. So Sanji was able to use them despite being swapped with Nami. However, that isn't to say they don't have any limits at all.
In the WCI arc, Luffy comments that Katakuri will deplete his Kenbunshou haki if he drags the fight long enough.
Now as for Luffy's drawbacks specific to Gear 4th, it is because of overuse of the Busoshoku Haki. As mentioned on the wiki page:
When utilizing Gear Fourth, Luffy makes extensive usage of his Busoshoku Haki by coating his arms, legs and much of his torso in it to the point that they turn black from hardening.
Combined with Luffy's rubbery composition and his compressed muscle structure, Gear Fourth both adds explosive power to Luffy's attacks and grants him enhanced defensive capabilities far beyond what he can demonstrate with or without Gear Second or Gear Third
The cause for Gear 4th explosive power is not the Haki alone, but how Luffy compresses his rubber body. Similar to how he increases his speed by pumping his blood faster in Gear 2nd. This means his physical body is under a lot of strain, as Rayleigh mentions when Luffy showcases these abilities to him.
Once this limit is reached, Gear Fourth automatically deactivates, leaving Luffy exhausted and barely able to move. He is also left unable to use Haki for ten minutes after using this technique as it quickly depletes his Haki reserves. The technique also greatly taxes Luffy's stamina, as using it just once leaves Luffy so physically drained in the aftermath that he can barely stand and fight, even after he has regained his Haki and mobility.
Akin to Gear Second, using Gear Fourth causes Luffy's metabolism to increase rapidly. In fact, after Luffy stuffed himself to a bloated size in his fight with Charlotte Cracker, using Gear Fourth once made him burn off the massive amount he had eaten in a manner of minutes
The exhaustion and side effects are so great that he can't properly stand and fight even after regaining Haki and mobility. However, it should be noted that Luffy's ability to bear these drawbacks, or maybe reduce their effect, is improving as seen during his fight with Katakuri where he manages to run around and dodge a bit even after his first attempt fails.
