When Blackbeard touches a devil fruit user, it is as if they don't have powers. Unlike the sea, when he drains your power, even physical property changes to your body are cancelled out.
The Wikia on Yami Yami no Mi talks about this:
This was shown when Luffy's stretching powers were canceled on touch, unlike the effect of a body of water or seastone, both of which just inhibit a person from controlling their powers. It was shown earlier when Luffy was stuck underwater that his neck could still be stretched above the water's surface, and that Busoushoku Haki would simply nullify the defensive aspects of a Devil Fruit such as intangibility and the malleability of rubber.
The episode they are talking about is episode 447. In this episode, Blackbeard uses vortex to pull in Luffy and smashes Luffy into the ground. Crocodile notes that there was blood. This is strange since Luffy, being rubber, should be immune to the physical damage caused by being slammed into the ground.
Another use of the Yami Yami no Mi allowed Blackbeard to permanently remove a devil fruit user's power and give it to himself. When Whitebeard died his power was still active in Blackbeard, thus causing the fruit to not regrow. This is proof of the permanent transference of power.
In either of these cases, either the case where the devil fruit user has their power drained temporarily by touch, or permanently like in the case of whitebeard, would the drained user regain their ability to swim?