Some feel they're being shut out of what could be an economic boom for the Black community