An agent does not only answer questions. It sends messages, writes files, and changes records in the apps you connect to it. Permissions are how you decide in advance which of those actions it may take on its own, and which ones it has to check with you first.
How permissions work