Not to take the side of your neighbor, who is obviously a lunatic, but what are four or five year old kids doing playing baseball outside without parental supervision?
My neighbor has a four-year-old and a two-year-old and they are outside by themselves all the time. The older one comes over and I have had to tell him one a weekly basis not to come in my gate and play on my kids' swing set and trampoline without an adult's permission. I've got no problem with him coming over and playing, but I don't want him to get hurt on my property when I'm not even home and his own mom has no idea where he is.
Also, if he's outside and sees me pull into the driveway, he'll deliberately run into my yard and into the path of my car because he wants to see if my kids are in the car with me. He wants to play, and if I tell him no (which is usually, because for pete's sake, we just got home from somewhere) he wants to know why, and where we were, or whatever. The same if he sees me leaving -- where are we going? When will we be back? I brush him off, usually, and I feel bad doing it, but I've got 99 problems and your inquisitive ass ain't one.
I live in fear that he's going to turn up behind my van while I'm backing up one day and it's going to be tragic. But his mom is hopeless. I've known her since high school. The kids live with their single mom and her parents, and between the three adults in that house nobody really seems to care where the kids are or what they're doing. All I can do is be extra vigilant about backing into or out of my driveway and keep telling the kids to ask their mom or grandma before they come into my yard.