I use the number system in several ways. I always take it as a granted, though, that the minor/major/dim. relationship applies even if the roman numerals are not used. If I am asked to play, say, a three, I would play some sort of minor chord on the three of the scale being played.
I use the number system to transpose chords. If I am playing a ,say 1-3-5-b7 on a C and I want to move that to a F, I just play a 1-3-5-b7 on F. Tell me to play a 7 in the key of C, I will give you a B dim(unless my ear said some other B).
It is hard to always tell what someone is playing by looking at their right hand, because often they will be playing a full chord in that hand. When placed with the bass note, it is another chord. Example, If you play an Emin7 with your right hand, and a C with your left, you are actually playing a Cmaj9th. You see some weird chords being played with the right hand in 7-3-6s.
In any of the number systems, the most important thing is to know the number of each note of the scale in any key(IMHO).