Can somebody remove my pass from their phone?

Yes. They delete it in Apple Wallet or Google Wallet like any other pass, and your messages and updates stop reaching them. Every pass also says how on its back, in the row Stopping this pass.

On your side, Active Passes on the Wallet screen counts the passes still on phones, as far as Apple and Google let us know. When and how a phone lets us know a pass has gone is up to Apple and Google.

We never learn who removed it. A pass carries no name, phone number or email of the person who kept it. See Does the pass tell you where somebody is?.

For the person holding the pass, the steps are in Removing a pass from your wallet.