Stafford was Air Force and Conrad Navy. While, there has always been talk that Armstrong got the nod because he was civilian, Deke Slayton also said anyone could fly any mission any time.
Also, it seemed like astronauts tended to think John Glenn would have commanded 11 had JFK not effectively grounded because of his popularity; then, in lieu of Glenn, Grissom was favored. Of course he died the Apollo 1 fire.
I have no doubt Armstrong’s civilian status came into play at some point, as did so many other factors.