Neil Oliver: Government leaders are frightened of their own people – and there’s a lot of us