It would create chaos and it might even ruin Doflamingo's scheme. Civilians don't remember their lost loved ones, but the toys remember everything. So you would expect the toys to try and remind their loved ones of who they are as we have seen in chapter 717 when the toy supposedly malfunctioned suffering of human disease.
Humans might remember their loved ones or might start to dislike the toys, which would disrupt the peaceful kingdom Doflamingo had built. Now imagine if the toys would be able to spend intimate alone time with regular humans, this would mean situations like these would happen more often. This could then go either of two ways, none of which would be beneficial for Doflamingo. Either the human would not recognize their lost loved one and freak out as in chapter 717, creating chaos. Humans might start to loath the toys and possible retaliation might happen. The other possibility is that the human in some way might even remember who the toy was and Doflamingo's schemes would be discovered. So the more control Doflamingo holds on the toys, the less chances he has of being found out.
If toys would be able to visit houses, they would be able to plot schemes to overthrow Doflamingo and possibly kill or harm Sugar, which they would have a real hard time doing if they would be watched constantly. Doflamingo could have trusted Violet's ability to oversee any strange activity, but how would you be able to trust the daughter of the former king with such a big responsibility. And even if he would, she wouldn't be able to oversee every toy at every second of the day. Her powers should be used more usefully elsewhere anyway.
Additionally, if toys were free to go about as they liked, they could travel to other islands and expose Doflamingo's scheme. Keep in mind that they were the only ones aware of Sugar's ability apart from the Donquixote Pirates. I doubt the government would ignore the cry of help of a bunch of toys who claim Doflamingo had made them like this. Especially since coincidentally Doflamingo's island is the only one having living toys. Don't forget that among the toys there are also Government officials, who would know too much about the inner workings of the World Government for it to be just a strange malfunctioning toy, who just happens to know too much.
Toy Soldier
As mentioned by OP in a comment, Toy Soldier has been seen in the private companionship of Rebecca. He was seen training her for the hard future to come to prevent her from being harmed once again. Since this is a special case and since it was only possible by plot armor a great deal of luck, I will make this a separate part of the answer. I want to point out that Doflamingo's level of dictatorial control is so high (and quite praiseworthy to be honest), that it really needed this much of luck for it to work. So now how was the toy soldier possible to break the law?
As we know, every toy is put to a contract by Sugar, right after transformations. The contract states only two simple rules, which can be see in chapter 737 when Cavendish became a soldier.
- I will not bring harm to humans.
- I will bow to the orders of the (Donquixote Pirates') family.
Two simple rules that make the toys unable to disobey any law put out by Doflamingo, which include not entering houses, and going to the toy factory every night. Toy Soldier on the other hand, was the first and only person to become a contract-less toy as Sugar had forgotten to establish the contract as mentioned by Leo in chapter 739. Therefore he could disobey the laws as he pleased. This didn't bring him out of the woods just yet though. He knew Viola had the giro-giro (aka glare-glare) fruit, and he knew she had forgotten about who he was. If he would disobey the law at this point, he would have been caught regardless. That's why in order not to stand out to other humans and especially Viola, he obeyed the laws with the other toys and went to the toy house every night as he mentions in chapter 721. This all lasted until Rebecca was kidnapped.
At this point (still chapter 721) he couldn't just sit back and let it all happen, so he rescued her regardless of the laws. This made him a criminal and a fugitive from that point onwards. Even if he would obey the laws now, he would be hunted down by the police as we have seen when the Strawhats first arrive at the Colosseum in chapter 703. Yet he is too badass to be caught by mere mortals. Doflamingo could have tracked him down, but he probably didn't care or might not even have been aware of the situation. Viola on the other hand, knew well about everything that had gone down, but by now she trusted Toy Soldier and wanted him to keep looking out for her niece Rebecca. She mentioned in chapter 740 that she knew all about him and Rebecca, but that Viola kept it a secret for Doflamingo. It was also from this point onwards that he was seen in the private companionship of Rebecca. It was only after he became a fugitive that he started breaking the laws. He didn't have much choice.
To conclude, Toy Soldier confirms what I wrote as the original answer. Doflamingo wanted to keep his peaceful regime as long as possible and he needed laws for that. The negative effect that made Doflamingo implement that law, was to prevent what Toy Soldier had done from happening. Remember that if it wasn't for Toy Soldier, nobody would have known about Sugar's ability and the toys would never have been freed, because nobody would have known about it. So by Sugar's neglectance her secret got out and he had gathered allies to overthrow Doflamingo.
It took him 10 years to do so, but here he is back to his human form and riding towards Diamante and Doflamingo to avenge his wife and restore king Riku's honor.