Selfish mining is a concept that was elaborated on in detail by Cornell researchers in a 2013 paper titled Majority is not Enough: Bitcoin Mining is Vulnerable. The authors propose that the mining incentives, designed to keep Bitcoin miners honest, can be gamed and are therefore broken.
By selfish mining, a miner (or group of miners) increases their revenue by strategically withholding and releasing blocks to the network. Typically, we expect a miner to announce a block as soon as they find it. If the block is confirmed, they will get the block reward.
By not broadcasting their block right away, though, these miners effectively create their own private branch of the blockchain. The rest of the network continues to build on the previous block, while the selfish miner builds on top of this new chain. From that point, both chains will look completely different.
The goal of the selfish miner is to always remain at least one block ahead of the rest of the network. Nodes accept the chain with the most accumulated proof of work as the valid blockchain. At any time, the selfish miner can reveal their chain. If it is longer than the one followed by the rest of the network, the existing blocks will be discarded, and transactions reversed. The miner collects all of the rewards from these blocks and causes other parties to waste resources.
The success of selfish mining is dependent somewhat on luck, but mainly on hashing power available to the miner (also known as hash rate). The authors of the paper note that, as miners on the public chain do not want to waste resources, they will join selfish miners on the alternative chain. In the long run, selfish mining could undermine the decentralization of Bitcoin, as it concentrates hashing power into smaller pools.
On the surface, selfish mining seems lucrative. However, many are skeptical of its ability to severely impact Bitcoin. By centralizing the network in this manner, participants would be fundamentally removing the value proposition of the cryptocurrency. For them to be profitable, it is in the best interest of miners to act honestly. After all, their profit is denominated entirely in Bitcoin.