I don't know if I should ask my Sifu about breaking boards and/or bricks..?

Boards do hit back! when you hit them wrong. Breaking is a part of martial arts and helps you focus for street fighting. Wearing gloves all the time will throw off your focus. Even Mike Tyson hurt his hand twice in his street fights from lack of focus. Even Bruce lee practice breaking.