I've played 4 a good bit.
It has some good points, it has some bad points. But I do tend to think that, gameplay-wise, it is an improvement.
Granted, the Sims can have odd behavior glitches at points. Some of which I dont remember being in older games. For instance, since you mentioned toddlers, I remember the last time I was playing it, the family consisted of 2 parents and their toddler child. But there was a specific wonky AI thing that kept happening. In particular the parents would get all weird once the toddler was put in a high chair (you know, for meal time, one of those chairs). They would pick her up and put her in it just fine, but then would just... not feed her and they'd get all glitchy when given direct commands to do so. It's that sort of glitch where you tell them to do something, they'll walk over to where the action should happen, and then they do that really awkward "stop and stare" thing that they do before randomly wandering off. You've probably seen them do this before in the earlier games, it's a common glitch type. But it was really consistent for this particular action for who knows what reason. But they would take her back out of the chair when given that command no problem (so the solution was not using the chair when feeding her, which just bothers me). (I think that's been patched though)
I tell ya, it probably sounded a bit odd for family in other parts of my house to randomly hear me yell "JUST FEED THE TODDLER, YOU COMPLETE LUMP!!!" after getting a tad frustrated with this. Yes, I get a bit loud when gaming at times. I'm not proud of it. But hey, at least I do admit to it.
But if you can put up with Sim AI occasionally getting loopy, you'll probably like it (and the overall gameplay experience, and performance, is WAY smoother). And yes, they do have way more options for toddlers/kids in this game compared to the old ones. Like, WAY more. Toddlers/kids have a ton of actions & activities they can take, and drastically improved AI for themselves, and more interactions with older Sims, and there are a lot of different objects and furniture and stuff for them. Though, if you want extra focus on toddlers, you might want to browse through some of the expansions and stuff packs. The one big problem I and many others had with Sims 4 is that the base game by itself is a bit barren in terms of *amount* of content (though this surprised nobody at all). So yeah, have a look around at them, so you can pick ones that will enhance the gameplay elements you find most interesting (or that add mechanics you might like). The good thing is that there is a LOT to pick from at this point, both full expansions and also the smaller "stuff packs".
It's certainly an improvement over 2, at least in my eyes.
The comparison to 3 is different though. The #1 thing that bothered me (and apparently most fans) about 4, aside from the low starting content part, is that 4 ditches the open-world neighborhood system that 3 uses. In 3, the entire neighborhood is loaded in as one single giant area, and your sims can freely roam around anywhere in it, even in spaces where there isnt really all that much there. But 4 uses a very restrictive closed lot system, similar to how the very first game did. So there isnt anything between lots except loading screens, and individual neighborhoods dont have many lots for some reason. But, you can freely transport from one lot to another easily, so your sims can go to the park or whatever and there will be lots of things to do there (and you can edit those lots if you want). Still, that part is quite the bummer. I have no idea what they were thinking in making that change, NOBODY liked that change.
Well, it's possible they made that change due to the fact that 3 tended to eat your PCs memory. LOTS to load in at once. I imagine though that modern PCs would have far less trouble running it than PCs at the time of its release did.
But in terms of actual actions and things your sims can do? 4 beats 3 out. Particularly in terms of interactions with other sims. 4 specializes in that quite a bit, 3 wasnt nearly as good at that part (and the first two games cant compare to 3 or 4 at all).
Dont get me wrong, the first two games are still good, but yeah, 4 is going to be a huge improvement, though 3 can also be an option depending on what you're after. But since you're after an enhanced family aspect, 4 is probably the way to go, definitely. Just... yeah, expansions.
If you want to know about the expansions, there's a channel on Youtube called LGR who reviews like, all of them. A lot of detail, so you can know whether a given expansion is actually what you want before buying.
Does that help? Or did I not quite answer what you were looking for?