The KJV translation of verse 24 is closer to the literal meaning of the Greek text:
24 saying, “Let us alone! What have we to do with You, Jesus of Nazareth? Did You come to destroy us? I know who You are—the Holy One of God!”
The phrase “what are we to you” is from a Hebrew idiom best translated as above in the NASB, “what business have we with each other?” The question of authority is specifically being raised here, and it’s being asked by someone with equivalent authority – a peer to Jesus of Gamla, High Priest. Possibly a local rebellion leader, perhaps another high priest. Perhaps the point of having Jesus teach in this synagogue was to instruct, to redirect, this particular person.
“I know who you are” – This anonymous rebel was having none of it, and refused to recognize the jurisdiction of Jesus in Capernaum. “You are not the boss of me,” he says. He recognizes Jesus as an existential threat, naming him a Nazirite and a “Holy One of God”. Jesus is not anonymous, he is known by name, by face, by reputation. As high priest, he would have been known as a Pharisee – Capurnaum was a Zealot stronghold full of rebellious proles. He would likely have been both known and seen as a threat simply due to his associations.
“Have you come to destroy us?” – This is a lot to unpack. On the surface level, the Pharisee v Zealot conflict was being indicated, with a Pharisee high priest teaching in the synagogue of a Zealot hotbed indicative of being ‘corrected’. On another level, the sectarian violence that made Jerusalem a pit of misery could be indicated. Inasmuch as Jesus represented the coming Roman Emperor, he was, in fact, coming to destroy these people specifically.
For the second time, the character Jesus is identified as a Nazirite. This time that identification is spelled out, saying that he is a “Holy One of God”. Like Samson and Samuel, Jesus is seen as a vowed dedicant for God. However, given the balkanized politics of the region, there was a lot of argument over authority. His status was recognized, but they disagreed over who should be in charge of the Zealots.
Two Jesus theory: the man with the “unclean spirit” is Shy Jesus, and this interaction represents the conflict of authority between Rome and the Judean Temple. Shy Jesus recognizes that Anointed Jesus has greater authority and submits.