Yes. It is possible for a program to execute an instruction that upgrades itself. There are three requirements:
- a PDA of the program is the program's upgrade authority.
- the program invokes the BPF upgradeable loader's deploy instruction.
- the program has access to a new version of the program.
An example is the SPL governance program. A proposal to upgrade the program, once passed, would enable a program upgrade of the governance program itself, assuming it references a pre-existing program buffer. To execute the upgrade, a user would have to submit an instruction to spl governance telling it to execute the proposal.
The program's upgrade authority needs to be set to some PDA of that program. For SPL governance, it would be the authority PDA for a particular governance within a realm. This would permit the program to CPI the BPF upgradeable loader with an instruction to upgrade itself by calling invoke_signed with the seeds for that PDA. The program needs to be implemented such that it can receive an instruction that causes it to invoke the loader to upgrade itself. The SPL governance program allows this because it will execute any arbitrary instruction that passes a vote.
There's also a question of where the new version of the program comes from. Somehow, you need to provide the binary program data to the BPF upgradeable loader to upgrade a program. The standard approach would be for a user to upload the data using the cli with solana program write-buffer. Then the program could execute the deploy instruction to upgrade itself with that buffer. But the possibilities are endless here. The custom program can take any arbitrary data from another account or instruction data (potentially requiring many transactions to upload large amounts of data on chain), and load it into a buffer account on its own before telling the bpf upgradeable loader to deploy the buffer.