In the Military, there are three units, each with its own patch: Scouting Legion, Military Police and Stationary Guard.
Left to Right: Scouting Legion, Military Police and Stationary Guard
So far in the anime (up to episode 12), from these three we've seen (if I didn't miss any) only the Stationary Guard's patch.
Most of the main characters (Eren Jaeger, Mikasa Ackerman, Armin Arlert, Jean Kirschtein and others) are in the Scouting Legion. However, they seem to have a different patch that exhibits two crossed swords. They've been wearing this patch since (in episode 2) formation, and the people in charge of training then were also wearing this patch.
Even characters, such as Annie Leonhardt, that belong to the Military Police exhibit the Crossed Swords patch.
What differs in the anime from the manga?
Were both the patches and the character's military affiliations changed?