In the Punk Hazard Arc, Zoro has a fight with a logia named Monet. She had the snow snow no mi power. In that fight Zoro defeated Monet not using haki but using some form of fear relating to an animal.
How exactly did zoro defeat Monet using this new ability of his? What this ability of his do? Have we seen the same ability like this being used by someone else?
If you look at the wiki page of Monet, the fight is very clearly explained in there (emphasis mine).
Sanji, Tashigi, and the G-5 Marines suddenly arrive at the Biscuits Room while Monet and Zoro are dueling. The Marines compliment on Monet's beauty, causing her to blush, and hiding her blush with her wings, and buckling her legs, despite the serious situation. After realizing that they are also attempting to get the children, Monet transforms into her demonic body again and attacks the Marines out of fury, biting off a Marine's shoulder. Tashigi then slices her with a Haki-imbued attack, injuring her. She then declares that she will stay and fight Monet.
While Tashigi and Zoro argue, Monet tries to use a sneak attack on Tashigi and gets the upper hand due to her speed. After Tashigi easily dodges her "Snow Rabbits" attack using Soru, Monet opts to blind her with her blizzard and then sinks her teeth into Tashigi's shoulder, trying to tear it off as she did to one of the Marines. However, Tashigi gets a hold of her, making her unable to tear her shoulder off. As Monet starts to bite harder on Tashigi's shoulder, Zoro cuts the harpy in the cheek, making her let go of Tashigi. She is surprised at first because she assumed Zoro would never attack women, but gets outright frozen in fear as she sees the swordsman rush towards her. Monet is then sliced vertically in half by Zoro. However, Zoro did not imbue Haki into his attack, allowing Monet to survive. Due to realizing what would have happened had he used Haki however, she is left terrified to the point where she can hardly put her body back together. As Zoro walks away, Monet manages to partially reform and attempts to stab him in the back with her ice picks, but Tashigi quickly deals the finishing blow. Monet then falls to the ground, defeated.
Thus to answer your question, How exactly did Zoro defeat Monet using this new ability of his?, there is no new ability/technique which Zoro used against Monet. It was just that Monet was under the belief that Zoro would not attack women (as he was just defending against her and not attacking when trying to save Nami, Robin, and Chopper). But suddenly, when she sees him attack, she's stunned that she was wrong and what would have happened had Zoro used Haki in that attack. That leaves her in a terrified state so much so that she is not able to get a grip of herself for a while.
Zoro did not defeat Monet. The attack involved no haki.
Acoording to the story, Monet is first sliced vertically in half by Zoro. However, Zoro did not imbue Haki into his attack, allowing Monet to survive. Realizing what would have happened had Zoro used Haki, she is left terrified to the point where she could hardly put her body back together.
As Zoro walks away, Monet manages to partially reform and attempts to stab him in the back with her ice picks, but Tashigi quickly deals the finishing blow. Monet then falls to the ground, defeated. When the poisonous gas starts flowing into the Biscuits Room, Zoro and Tashigi flee and Monet is left behind.
Later we see that she manages to escape to a different section of the facility. While amongst the crumbling rubble of the lab, she contacts Joker and apologizes for her failure and came to the decision to set off the remaining weapon to destroy the entire island and kill everyone on it.
