Before the first Galician manor house—the Torres do Allo—existed, there was a great castle nearby, at the top of a hill. Today nothing remains aside from stones that are being studied by archaeologists to determine its position and exact age. However, the name of the mountain, called ‘big castle,’ already gives us a clue that it was precisely here that this fortification of the House of the Altamira was located.

In 1455 the counts Lope Sánchez de Ulloa y Moscoso y Aldonza de Mendoza, sister of the archbishop of Santiago de Compostela, founded the County of Altamira. It is one of the most powerful families of the Galician nobility, owners of several fortresses, houses and castles, as well as land and fiefdoms since the 9th century. But in 1467 they would see their patrimony threatened by the Irmandiño revolts, the greatest European social revolution of the fifteenth century. The constant abuses of the great noble houses ended up provoking the peasant uprising that would fight against the landowners by toppling their towers and castles. The Altamira Towers and Castle of Allo are among the 130 strongholds believed to have been destroyed by brotherhoods.

Although the Irmandiños were defeated in 1469, and many fortifications were ordered to to rebuilt—going so far as to use the labor of the insurrectionists themselves—the Castle of Allo would not last much longer. The Catholic Monarchs, who saw their authority threatened by the great houses of the Galician nobility, ordered all fortresses to be destroyed. The documents say that the demolition of the Castle of Allo was ordered to be carried out around 1487. However the Altamira family implemented an impeccable strategic move to continue maintaining its influence under the new crown.

The Counts of Altamira not only accepted the destruction of the Castle of Allo, but also moved their residence to Madrid in order to be closer to the Royal Court. But in the lands that now belong to the City of Zas, they would leave one of their daughters as the owner of Allo to continue governing the region. Berenguela Sánchez de Moscoso married the one who would become the first of the Riobóo: Gómes de Riobóo Vilardefrancos, an important vassal of the House of Altamira, and surely the administrator of the castle in the absence of its owners. For her, they ordered the so-called ‘house of the Allo’ (referring to ‘ayo,’ servant of the noble houses) to be built below the plain, and then began the construction of what we now call the Torres do Allo, which would for fifteen generations be known as the House of the Riobóo.