The fallout is steep among convicted senator's old neighbors