(Note: I have just typed this all up and have once again found I am extremely wordy. You’ve just happened to hit on one of my ‘Strong Thoughts’ topics, so I am now subjecting you to my yapping. I am not a medical professional. I do support risk mitigation. I also love actual data!)
We absolutely should take precaution to keep our areas dust reduced and wet clean as much as possible. There is no truly “safe” amount of fine silica dust.
However, I went and snagged the work of others to help clarify what kind of risk we are actually talking about:
Reporting for 2017 found 20,000 cases of silicosis that year worldwide with 200 in the U.S. That includes many countries with lackluster (or no) protections and safe practice oversight. Occupations with the highest risk are those that cut/drill/grind materials like concrete, stone, sand and rock (mining, sandblasting, construction, ceramic production, countertop sanders, etc.). OSHA says about 2.3 million people in the U.S. alone are exposed to silica at work. How many of those 200 are studio potters? I don’t know, but statistically speaking, it’s likely zero, considering the level of exposure people have with large industrial operations in comparison.
So yeah, be safer than you technically need to be, I love that. But there was actually a really fabulous CDC study (NIOSH) done in 2007 with a studio in Cincinnati where they geared up four workers with detection equipment and had them go about two full workdays as usual. Including tossing around bags of dry clay and, the generally considered riskiest action, mixing glazes. Seven of the work days had zero exposure. There was a singular alert of detectable respirable silica (an employee dumping 50lb bags of dry clay into clay mixers), but it was at the allowable safe limit. A single ping, once. Neither the studio nor the glaze room had detectable issues. It all has to do with the very tiny particle size necessary to cause silicosis. Many other studios have also brought in detectors to monitor their risk levels, over and over, no issue.
But while we are at it though, if you want something to be worried about, I have much more exciting ones for you! Decomposing metal oxides are…fun times. And have you ever found yourself yourself standing right next to the kiln when it is firing? Yeah, try to limit that action, even with the vent hoods down. Has a crusty old 1970’s potter passed you a bucket of a beautiful satiny blue glaze? I bet you a shiny quarter the secret ingredient is barium. Powdered spodumene causes near immediate lung irritation, that can’t be good.
All said and done, you aren’t wrong to be cautious and want to take care of your lungs, you’ve only got the one set. Good practices benefit everyone. I’ll still wear a respirator if doing “riskier” things, but I’m not going to leave the room if someone does a quick sweep up of trimmings. Carcinogens from all the barbecue I eat are probably a higher threat. 
This is my fear, not a broom.
lol, need our emoji reactions back.
