Downtown Santa Fe, New Mexico The Old El Palacio Real Fortress and Castle, built by Order of the Spanish Crown 1610 to 1612. Seat of governmnet under three flags-Spanish, Mexican, American. From 1610 to 1910 the residence of over a hundred governors and captains general. This is the oldest public building in the United States.