Allergies tend to produce itching, bumps, redness, or other uncomfortable symptoms. If you’ve had exposure to latex before (gloves at the doctor’s office, balloons, etc.) and had no reaction, it may not be an allergy but rather an irritation. What you were feeling may have also been irritation from the friction of sex.
If you continue to be concerned about a latex allergy, you might try a polyurethane condom instead. They are available in the same aisle as latex condoms; just look for one that says ‘polyurethane’ instead. They can range from $6-$10 for a multi-pack.