Tales From A Lazy Fat DBA

Alter database VS Alter system

An “alter system” command is (mostly) only possible in status OPEN. The only exception from this I recall presently is  “alter system set param=value” to modify initialization parameters. That is already possible in NOMOUNT status.
“alter database” on the other hand is already possible in status MOUNT,

ALTER SYSTEM is an Instance Level command. Generally this applies for running processes, parameters etc. (however “ALTER SYSTEM DUMP” seems to be an exception).

ALTER DATABASE is a Database Level command. Generally, this applies to the physical structure of the database.

Think of an RAC environment. Most ALTER SYSTEM commands (ALTER SYSTEM DUMP is one exception) are local to the instance (although SET can set for multiple instances). The ALTER DATABASE commands are for the whole database.