I think the concept of a no-trade mechanic would be a complexity that would decrease the chances of this happening.
In regards to brick sellers, I see little impact.
I assume you would need permissions on a building to do this, so that negates people salvaging buildings from the wild.
For the person demolishing a wall to get say maybe 20% of the bricks back (and I do think it should be limited), this is the same person who would likely make their own bricks.
I would see this done in the minority of cases, perhaps someone adjusting an existing building, or someone who accidently built the wrong wall type.
Whereas someone like me, would still be buying bricks to build entirely new buildings and deeds. And in this scenario I would be starting from dirt without a collection of buildings to recycle.