This final phrase seems to indicate that the previous several verses of parables were sparked because the Pharisees said that he had an unclean spirit — that he was encouraging rebellion against temple rule.
To restate the previous three verses: rebellion against the temple (the rule of man) is forgivable, but rebellion against the holy spirit of Rome is not.