I have friends in Rye. There are a couple of pre-school groups, and swimming sessions for under fives in a lovely pool. The salts play area isn't bad, and there is a skate ramp and cricket/football club for older kids. It's not great for older teenagers tbh, the only nightclub is a bit of a meat market and the secondary school had quite a bad reputation for drugs etc. back in the 1990s, but that may well have changed. For adults, it's lovely. Lots of arty stuff going on, good pubs and restaurants, a great bonfire event every year, fantastic walks in the surrounding countryside. Property prices are a bit higher than nearby towns (Hastings etc) but I'd be quite happy to live in Rye.
I wouldn't let the drug issues put you off, tbh. Probably no worse than any other town, and it was mostly just kids having a spliff at lunchtime rather than heroin etc.! But as I say, that was in the 1990s and it has probably changed by now. Rye is great for most ages apart from those two or three years when you want to go a bit wild, go clubbing and meet lots of new young people. But you can pack them off to college/university to do that!!