The House of Alpin was named after King Alpin II of Dalriada. His son, Kenneth I brought together the crowns of the Scots and the Picts and became the first King of what with hindsight can be called the Kingdom of Scotland. In his own time he was called King of the Picts, or King of the Picts and Scots. From Donald II's reign the king was styled the King of Alba.
From the time of Donald I, succession within the House of Alpin followed the ancient Celtic law of tanistry, under which the extended family select a tanist, or successor, to the king. The key benefit of the system in a time when few men lived into old age were that the successor was usually an adult able to undertake the role: it is arguable that Scotland might have benefitted from such a system in later ages.
There have also been suggestions of an agreement existing between the two main arms of the House of Alpin that the crown should alternate between them: this is certainly pretty much what happened in practice.
But the drawbacks of the tanistry system came to outweigh the benefits. The most serious was the unfortunate habit of kings dying at the hands of their successors on the other side of the family: a tendency that might have been more constrained if succession had remained within the king's more immediate family. The overall impression is of a system that tolerated and even rewarded a degree of savagery that is surprising, even in the context of the end of the dark ages.
Tanistry's end was brought about, still more savagely, by Malcolm II. He tried to ensure that succession remained within his own branch of the House of Alpin by killing most of those who might have a claim from elsewhere within the family. He was not wholly successful, having overlooked Lulach, who went on to become the final king of the House of Alpin. But he did enough to ensure that succession found its way in turn to two of his grandsons, Duncan I and Macbeth: and to his great-grandson, Malcolm III Canmore, the first king of the House of Dunkeld or the House of Canmore.