Exactly, we are all victims of being sinners as no one is perfect.
Also remember perhaps that person repents frequently. "Do not take someone's silence as his pride, perhaps he is busy fighting with his self." — Ali ibn Abi Talib.
Last time I remember praying is a must for everyone regardless of how much of a sinner you are, even if you are making up for the missed Prayers, just keep trying as long as you're working towards praying 5 times a day.
Usually those who don't pray are likely to make this statement which makes no sense because not praying at all is classed as a sin - at least the latter is practicing the obligatory acts of worship so I don't understand this mentality.
Another reminder never let go of your prayers (don't loose hope) don't let such words dishearten you either - this may also be a test for you to see how strong your Imaan really is. As a result it will strengthen your relationship with Allah further more if you have dealt with it properly.
Allah says "Verily Salat restrains (oneself) from immorality and all that is forbidden" 29:45. So basically you don't want to let go of your Salah regardlessssS.