It happens to native speakers too. A few weeks ago I was on the playground with my daughter. I was speaking with another parent who was also from an English-speaking country. He used an English phrase I hadn’t heard before and I had no idea what he meant.
My solution was to say: “I’m sorry, I’m not familiar with that phrase. Could you explain what it means?” He did, and the conversation continued as usual.
Native speakers often use idioms in English which non-native speakers don’t understand. And as you can see, sometimes native speakers use English phrases that other native speakers don’t know.
If you don’t understand what someone has said, it’s okay to ask. (I do it too!)