My mom was telling me just recently a story about Brazil nuts being called that. I was saying how I thought it was awesome that back in the '50s, my grandfather (who was a minister) would often host traveling Black ministers in their home. At the time, even in Northwestern Ohio, they couldn't stay in most white-owned hotels (even if they could afford it) and most other white pastors wouldn't allow Black people to sleep in their homes, eat dinner with their families, etc. My grandfather was an exception, and apparently he took a lot of flak for it, especially since (and this almost makes me ill to say) he had four daughters.
But when I told Mom how I thought Grampa was unusually progressive considering his place and time, she said, "Yeah, that's true, but it's not like he never used the n-word or anything. I mean, he would never call a person that, but I remember once, a Black preacher came to stay with us, and we had these bowls of nuts out, and some of them were Brazil nuts. Well, my dad used to call them 'n-toes,' but not the letter 'N,' you know -- that word you can't say. So that's what we called them too, I didn't know that word could refer to a person, I thought it just meant that kind of nut. So I told my sister, "hey, give me one of those 'n-toes,' and my Mom went white and shooed us out of the room and said, 'Don't say that, don't say that!' It wasn't 'til I was in high school that I realized why."
Still pretty mild on the racism scale, especially for the 1950s, but it sure is interesting how times have changed. (For the better, but we ain't there yet.)