Hello there, I'm Tak@, a system integrator.

One of the most thought-provoking pieces of news I've encountered recently in the world of information systems is a research finding that suggests a short essay of just around 250 words can surprisingly accurately predict a child's future academic performance, cognitive abilities, and even educational outcomes.

Large language models predict cognition and education close to or better than genomics or expert assessment

As a parent with an 11-year-old child myself, this fact is something I simply can't overlook. I want to delve deeper into this intriguing paper.

AI's Power: Breaking Through the Wall of "Unpredictability"

Historically, predicting human psychology and life events has been extremely challenging. Especially for complex traits like academic performance or non-cognitive skills (such as "grit"), traditional social survey data has only been able to explain a small fraction of individual differences.

For example, a previous large-scale study, the "Fragile Families Challenge (FFC)," had a prediction accuracy of only about 20% for GPA and economic hardship. For non-cognitive traits like "grit," the accuracy was a mere 5%.

These results reinforced the idea that human lives are fundamentally "unpredictable." However, in recent years, the emergence of deep learning techniques in natural language processing (NLP), particularly "Transformer"-based language models, is changing this landscape.

I personally experience the evolution of this technology every day. AI's ability to understand and generate human-like language has opened new avenues for extracting hidden information from data that was previously invisible.

The Nature paper focused on a unique longitudinal dataset from the UK's National Child Development Study (NCDS) This study includes short essays, approximately 250 words long, written by participants at age 11 on the topic, "Imagine yourself at 25."

By analyzing these essays with AI, the researchers attempted to predict individual characteristics, which had been difficult with conventional survey data.

As someone who designs systems, I find it incredibly promising that such potential lies hidden in non-standard data that has been underutilized until now.

Past Text Analysis Limitations and AI's Leap Forward

Previous text-based prediction studies attempted to forecast individual personality, mental state, cognitive abilities, and academic performance, but the explained variance was limited to around 5-10%.

However, a recent study published in "Science Advances" analyzed 240,000 college application essays (about 1,400 words per person) and achieved high prediction accuracy: up to 49% for SAT composite scores and 16% for household income.

While this research indicated the potential of text data for prediction, its sample was limited to college applicants, making it highly selective and homogeneous. This raised questions about its generalizability to broader populations.

The Nature paper improved upon this approach by:

• Using shorter text samples: They used essays of only about 250 words.
• Employing a more representative sample: They used data from the "National Child Development Study (NCDS)," which tracked children born in the UK in 1958, providing a more diverse and less homogeneous sample.
• Leveraging state-of-the-art NLP techniques:
  • GPT-based Embeddings: They used the deep learning model "text-embedding-ada-002" to convert each essay into a 1536-dimensional numerical vector. This allowed them to capture the semantic importance of words.
  • Multifaceted Linguistic Metrics: They measured 534 computational linguistic metrics, including vocabulary diversity, sophistication, and sentiment analysis.
  • Readability Metrics and Error Rates: They also included 31 readability metrics, as well as grammatical and typographical error rates in their analysis.
• Adopting the "SuperLearner" framework: To maximize prediction accuracy, they used SuperLearner, an ensemble model that combines various machine learning algorithms (e.g., Extreme Gradient Boosting, Random Forest, Support Vector Machines). This improved prediction accuracy on holdout data (data not used for model training) while preventing overfitting.

These innovations enabled predictions that surpassed previous limitations. From a system design perspective, I am truly amazed by AI's ability to integrate such diverse data and extract its latent value.

The Surprising Details of Prediction Results: What Essays Reveal About the Future

The Nature paper compared LLM-based essay predictions with teacher assessments and genetic data (polygenic scores, PGS), and also examined the predictive power of models that combined these sources.

Predicting Cognitive Abilities and Non-Cognitive Traits

Let's look at the prediction results for cognitive abilities and non-cognitive traits at age 11. Prediction accuracy is indicated by the R^2_Holdout score (0 indicates prediction performance similar to the training data mean, and 1 indicates perfect prediction).

• Emphasis indicates the highest value in each category.

This table shows that AI predictions based on essays (NLP) demonstrate accuracy comparable to, or even surpassing, teacher assessments, especially in reading ability. Surprisingly, predictions based on genetic data had lower accuracy than the other two methods for these abilities and traits.

While the overall prediction accuracy for non-cognitive traits is lower, both essays and teacher assessments showed similar levels of accuracy, with teacher assessments being relatively more accurate for extraversion.

Furthermore, the Nature paper's model showed it could predict over 10% of the variance in "Big Five" personality traits (agreeableness and openness) at age 50.

This suggests that a short essay written at age 11 has the potential to reach into an individual's characteristics decades later. These results, even for a system designer like me, emphasize the "timeless" value that data can hold.

Predicting Educational Outcomes

Next, let's look at the most significant finding: the prediction of final educational attainment (at age 33).

For individual models, teacher assessment achieved an accuracy of 0.29, essay-based AI prediction (NLP) achieved 0.26, and genetic data achieved 0.19. However, an ensemble model combining all three sources of information achieved a remarkable R^2_Holdout of 0.38.

This 38% figure becomes even more significant when compared to traditional research:

• Compared to the FFC study's prediction accuracy of approximately 20% for GPA and economic hardship, the Nature paper's model nearly doubled that figure.
• The Nature paper's model significantly outperformed "parental education level," one of the most well-known sociological predictors, which only had an accuracy of 0.12 for predicting educational attainment.
• Biological factors like birth weight (0.01) and height (0.03) were also shown to be of little use in predicting educational attainment.

This means that by combining just one essay written by an 11-year-old child with teacher assessments and genetic information, we can paint a more concrete picture of what kind of education that child will receive in the future than any previous prediction model.

The information systems we work with ultimately aim to positively impact human behavior and society. I believe this level of predictive accuracy significantly expands those possibilities.

"Embeddings" Technology Driving Text Prediction

The Nature paper analyzed various text-based features (information) extracted from the essays and decomposed their contribution to prediction performance:

• Traditional readability metrics
• Grammatical and typographical error rates
• Advanced computational linguistic metrics (566 metrics related to vocabulary characteristics and sentiment analysis)
• 1536-dimensional deep learning-based embeddings

As a result, the comprehensive model using all text information showed a 5 to 10-fold improvement in predictive performance compared to using only essay length as a prediction benchmark. This clearly indicates that more complex information, beyond simple text length, contributes to prediction.

More importantly, the comprehensive model showed only marginal improvement compared to a model using "embeddings" alone.

This suggests that most of the information derived from the text data used in the Nature paper is encapsulated within these deep learning-based "embeddings."

The "embedding" technology, where AI converts text into numerical vectors, can capture the semantic relationships between words and sentences. It's as if it's numerically representing the "mind" or "intent" behind the words.

I believe that deeply exploring the mechanism of why these "embeddings" possess such powerful predictive capabilities will be a major theme for future research.

AI and Human Collaboration: Future Implications and Ethical Questions

These research findings challenge the conventional idea that human psychology and social outcomes are "unpredictable." However, this isn't a simple story of AI completely replacing human capabilities.

Complementarity of AI and Teacher Assessment

The Nature paper also showed that teacher assessments still maintain high predictive accuracy. In particular, for mathematical ability, non-verbal ability, and non-cognitive traits like extraversion, teacher assessments even outperformed AI predictions in some cases.

This indicates that teachers' years of experience and their ability to observe each student from multiple angles are extremely valuable sources of information.

I believe that AI predictions have the potential to be used as a "support tool" to reduce teachers' burden and enable more individualized educational support. For example, in admissions processes, combining AI-analyzed essay data with teacher assessments could create a fairer and more comprehensive evaluation system.

Ethical Challenges and Future Responsibility

However, the improved predictive power of AI also raises important ethical concerns. In the past, algorithmic bias has been a problem in predicting criminal recidivism and credit scoring systems, leading to unfair outcomes.

In education, the history of bias in testing is long, and this is deeply related to contemporary discussions about algorithmic bias and fairness.

If an 11-year-old's essay can predict their future, isn't there a risk that such predictions could unfairly limit an individual's potential or label them? As someone who develops and operates systems, and as a parent, I have strong concerns about this.

• Ensuring Transparency: It is essential to make the process as transparent as possible, showing how prediction models function and what information they use to make predictions.
• Identifying and Mitigating Bias: Continuous effort is needed to identify biases hidden in training data and mitigate their impact on prediction results.
• Appropriate Regulatory Frameworks: As prediction systems become integrated into all aspects of society, it's crucial to discuss and establish appropriate regulatory frameworks that define their usage and ethical guidelines.

The Nature paper's results are based on a specific sample of individuals born in the UK during a particular period, so it's unclear how generalizable these findings are to modern students or other countries. Furthermore, the causal relationship between the information in the essays and the predicted outcomes is not yet fully understood.

Beyond Prediction: Fostering Individual Potential

The fact that a short essay written by an 11-year-old can predict their future academic abilities and educational outcomes offers us deep insight. It's not just about the accuracy of the prediction.

We shouldn't simply follow a predicted "future." Instead, we should consider how to use this information to maximize an individual's potential.

For instance, if AI can suggest early learning difficulties or tendencies in specific abilities, it might be possible to connect that information to personalized educational programs or early interventions.

The Nature paper demonstrates the immeasurable potential value of data and AI.

However, implementing this power in society always comes with ethical responsibilities. Can AI's "predictions" become a powerful tool for us humans to better shape our own "futures"? I believe that depends on how we, as developers, educators, and society as a whole, confront this new technology.

If your 11-year-old essay could predict your future, would you want to know the results? And how would you use that information?

Even if my 11-year-old essay could predict my future, I'm confident that by believing in human potential and guiding my child appropriately as a parent, a wonderful future will surely unfold, without being constrained by the predictions.

Follow me!