Downtown Santa Fe, New Mexico The Old El Palacio Real Fortress and Castle built by order of the Spanish Crown 1610 to 1612. Seat of government under three flags-Spanish, Mexican, American. From 1610 to 1910 the residence of over a hundred governors and captains general. This is the oldest public building in the United States.