TXST digital history project uses AI to unlock centuries of Andean history

For generations, historians studying rural Indigenous communities in the Peruvian Andes have faced the same obstacle: an overwhelming abundance of handwritten records and no practical way to connect them. 

Catholic churches manually recorded most baptisms, marriages, and burials in leatherbound parish records during the Spanish colonial and modern national periods of the 18th to early 20th centuries. But the households and families linked through those records remained scattered across thousands of pages, making it nearly impossible to interpret the lives of ordinary people and their relatives over time. 

Now, a Texas State University research team is changing that with a project called “Sondondo Valley. Population and Territory in Rural Andean History.” 

Led at TXST by Dr. José Carlos de la Puente, professor in the Department of History, an interdisciplinary collaboration between historians, computer scientists, and engineers is developing new computational methods to connect thousands of historical data points. The project sits at the intersection of history, artificial intelligence, and data science—an emerging field known as digital history—and could transform how researchers study communities that have long been overlooked. 

“The Andes for me is a laboratory to understand how Spanish colonialism affected Indigenous populations,” says De la Puente, who grew up in Lima, Peru. “But it’s very difficult to reconstruct the everyday life of rural peoples living in these villages in a systematic fashion. The only exception to that is the church records. We’ve been using these records for a long time but, taken together, they are overwhelming for a single researcher. Our goal is to standardize this information and clean it up so a computer can help us answer historical questions.” 

The project began in 2019 when Peruvian researchers—funded by the French Institute of Andean Studies—digitized parish record books containing baptisms, marriages, and burials. Later, in 2023, the project team, including De la Puente, directed a group of student assistants at the University of California-Santa Barbara as they processed some of the records into a computer database.  

TXST is spearheading the next phase of the project with a $23,400 Research Enhancement Program grant from the university’s Division of Research, which points to the project as an example of its focus on digital humanities as an area of research impact. De la Puente is using the funds to develop computational programs that can interpret the records and make connections from thousands (and potentially millions) of individual records. 

He hired two TXST graduate students—Vamsi Krishna Samavedam and Sneha Tulshiram Nitnaware—to bring their expertise in data science, machine learning, and analytics to develop the programs needed to interpret the data.

“When you have a marriage record, for example, you have the bride, the groom, the parents on each side, sometimes the grandparents, plus the witnesses,” De la Puente says. “In one record, you can have 20 individuals.”

What’s more, the same individual may appear multiple times and in different roles throughout life—in a baptismal record, later in a marriage record, and eventually in a burial register—but the computer initially treats each appearance as a different person. The TXST team is tackling that challenge by teaching computers to determine, through probabilistic record linkage, when those separate entries belong to one individual.

By linking those records together, historians can trace the history of an individual and begin asking entirely new questions about migration, family relationships, disease, demographic change, and community formation across generations.

“When I saw this type of project, I thought, ‘I’ve never done this before. Why not try it?’” says Samavedam, a master’s student in computer science. “It’s a great opportunity.”

Samavedam’s work focuses on cleaning massive datasets—more than 1.5 million records—and developing machine-learning workflows capable of identifying matching historical records.

“When they search for one particular person, they should be able to see that person’s complete history—the baptism, marriage, burial, everything—in one timeline,” he says.

Nitnaware, who is pursuing a master's degree in industrial engineering with a concentration in data analytics, says their programming must overcome inconsistencies in spelling, handwriting, and transcription.

“You can’t match historical records with pure pattern recognition,” she says. “You need to understand temporal feasibility and genealogical constraints.”

For example, a woman appearing in a marriage record and a burial record 80 years apart is almost certainly two different people with the same name.

To solve this problem, the students are applying probabilistic record linkage using Splink, a Python library that computes match probabilities through statistical inference rather than rigid rule-based systems.

“The system learns patterns from imperfect data,” Nitnaware says. “It understands that ‘Juan de la Cruz’ and ‘Juan dela Crús’ are likely the same person, even across decades and different record types in a baptism, a marriage, a burial.”

With continuing refinements based on De la Puente’s expertise with these records and native Andean rural societies in general, the model has been able to identify the same individuals across records with reliable accuracy.

While the project is rooted in the history of one region of Peru, De la Puente says its potential extends beyond the Andes. “We want to create something that can be replicated, adopted, and used by other researchers,” he says. “We’re really opening up a lot of new information.”

For example, De la Puente says he has visited with descendants of Native Americans who lived in the San Antonio Missions about applying the model to their records to gain new insights into their family histories.

The team’s long-term vision includes an open-access website where researchers, educators, and the public can search individuals, places, surnames, and historical events. Interactive maps would visualize migration across the Andes, while network diagrams would illustrate family relationships spanning generations.

De la Puente says he’s excited to unlock the potential of the new website in his research about the formation of Indigenous Andean communities.

“We’re going to create a website that is going to include not only the original images of the documents, but it’s going to have a search engine,” De la Puente says. “Then we can start asking the big historical questions.”