RESEARCH DOSSIER
Algorithmic Psychological Assessment and Predictive Modeling
Abstract. Algorithmic psychological assessment and predictive modeling use statistical and computational procedures to estimate specified psychological attributes or outcomes from behavioral, linguistic, psychometric, and multimodal observations. This fundamental critical narrative research monograph reconstructs the historical relationship between actuarial prediction, psychological testing, and contemporary machine learning. It examines the distinction between operational prediction targets and latent psychological constructs, feature provenance, statistical-learning models, information leakage, resampling and external validation, discrimination and calibration, and uncertainty. Original studies of social-media language, digital footprints, and smartphone behavior are evaluated alongside clinical outcome prediction and evidence of failures in cross-trial transport and proxy-based target selection. The synthesis identifies necessary distinctions among predictive association, construct interpretation, causal explanation, and beneficial decisions. It evaluates subgroup error, longitudinal drift, privacy, interpretability, and human–artificial decision processes while identifying open questions about independent replication, measurement invariance, prospective evaluation, and model governance. The research is a theory-guided critical narrative synthesis of primary and methodological literature, not an original experiment, protocol-registered systematic review, or meta-analysis.
Keywords: algorithmic psychological assessment; predictive modeling; machine learning in psychology; psychometric validity; digital phenotyping; personality prediction; language analytics; behavioral traces; actuarial prediction; external validation; cross-validation; data leakage; calibration; generalizability; algorithmic fairness; explainability; clinical prediction; human–artificial decision support; Artificial Era
The Research Object and the Logic of Algorithmic Assessment
Algorithmic psychological assessment is the estimation of an explicitly specified psychological construct, behavior, or outcome from observed predictors by a reproducible computational rule. Predictive modeling concerns performance on observations genuinely independent of model development. The research object includes what is to be predicted, when the prediction is made, which data are legitimately available, what comparison model is appropriate, and what interpretive or institutional action might follow. A classification of questionnaire responses is a prediction of those responses, even when researchers hope the response score represents a latent personality attribute. The long-standing account of construct interpretation by Cronbach & Meehl (1955) and the testing requirements of AERA, APA, & NCME (2014) make this distinction central to every empirical claim.
A psychologically relevant algorithm can combine self-reports, structured tests, language, smartphone logs, task behavior, or contextual information. Such inputs are indicators generated by social and technical circumstances rather than transparent windows into mental life. A model can learn distinctive lexical choices and nevertheless fail to measure the intended attribute outside its training environment. Flake & Fried (2020) describes how inadequate transparency about measurement decisions undermines cumulative evidence; the same problem becomes more consequential when automated processing creates an illusion of objectivity. An assessment claim consequently requires a construct specification, an account of indicator provenance, evidence for the criterion, and tests of how score meaning changes across settings.
The guiding questions of this critical narrative monograph concern the historical transition from actuarial judgments to data-intensive models; the relation between psychometric measurement and prediction; the design of independent validation; the stability of predictions across persons, institutions, and time; and the interpretability, fairness, and consequences of applying model outputs. The synthesis draws upon original empirical studies, psychometric theory, methodological investigations, systematic or scoping reviews where available, and official standards. It reports no newly collected observations and is not a protocol-registered systematic review. Yarkoni & Westfall (2017) provides the case for studying out-of-sample prediction as a serious psychological research problem alongside explanatory inference.
From Actuarial Judgment to Computational Prediction
Formal prediction predates contemporary machine learning. In Meehl (1954), clinical and statistical prediction were examined as alternative ways of combining available evidence. That distinction concerns the rule used to aggregate information; it does not require that the data themselves come from particular tests or interviews. A clinician's assessment may produce valuable predictors, while an explicit actuarial rule may combine them consistently. Later work by Dawes et al. (1989) developed the implications for judgments about human behavior, prognosis, and decision making. These investigations provide the historical foundation for algorithmic psychology because they treat predictive accuracy as an empirical property that can be measured against the same independently observed outcome.
The meta-analysis by Grove et al. (2000) compared clinical and mechanical combination procedures across studies of health and behavior. Its overall finding favored mechanical prediction, although the size and relevance of any advantage depend on the tasks, samples, and comparator conditions examined. The result supports a useful methodological principle: a formal rule must be evaluated against a genuine human-judgment baseline rather than assumed superior because it is mathematical. Mechanical combination includes straightforward scoring and regression rules; sophisticated neural models are not necessary to establish repeatable decision criteria. Historical comparisons also cannot be transported automatically to every contemporary clinical or organizational application.
The later machine-learning literature expanded the available functions, representations, and validation methods. Hastie et al. (2009) integrates statistical learning methods such as penalized regression, decision trees, ensembles, and support vector machines within a general framework for data-driven prediction. Yarkoni & Westfall (2017) emphasized that psychological explanation and prospective prediction answer different questions and proposed more explicit evaluation against unseen observations. The intellectual continuity is therefore clear: reproducibility of the decision rule, predictive criteria, and independent assessment were established before digital trace datasets. Contemporary computational capacity changes the scale of inputs and functions, while the central requirement of independent empirical validation remains unchanged.
Psychological Constructs, Scores, and Observable Targets
A target variable may be a questionnaire total, a clinician-assigned category, a school performance outcome, a behavioral task response, or a later event such as treatment discontinuation. These targets have different epistemic status. Self-report responses summarize what respondents endorsed under particular conditions; clinician ratings may incorporate interview conventions and available context; administrative records depend on organizational practices. Cronbach & Meehl (1955) argued for evaluating constructs within networks of relationships, while AERA, APA, & NCME (2014) treats the proposed interpretation and use of scores as objects requiring evidence. Machine learning can approximate a recorded label accurately without resolving whether the label is a valid representation of the psychological property named in the study title.
Criterion selection is therefore a scientific design decision. If the target is current reported distress, predictions should be described as predictions of reported distress; if the target is subsequent service use, predictive success concerns that future event, even when the investigators are interested in underlying psychological need. The case examined by Obermeyer et al. (2019) illustrates the serious consequences of substituting a convenient institutional proxy for a meaningful health-related need. Its setting is health management, but the mechanism generalizes as a warning about label choice in educational, employment, and psychological applications: an accurate model of an unjust or distorted target can replicate the distortion.
Construct validity is neither supplied by feature abundance nor reduced to a coefficient of agreement. Appropriate evidence may involve relationships with independent measures, discriminant tests, response processes, temporal stability, and consequences of score use. When an instrument is adapted to a social platform or language model, the response situation changes, so validation arguments must be reconsidered. Flake & Fried (2020) describes questionable measurement practices that impair interpretation; Park et al. (2015) offers a stronger empirical pattern by comparing language-derived personality estimates with self-reports, informant reports, and external correlates. Even such convergent evidence should be interpreted within the population and data collection circumstances actually studied.
Prediction Tasks, Estimands, and Decision Horizons
Every model has a target population, an index time, a prediction horizon, and a unit of analysis. Predicting an individual's questionnaire score from language already written before a survey differs fundamentally from predicting a later depressive episode from material observed in a defined preceding period. The first is primarily concurrent estimation, while the second asks about prospective risk and must account for outcome incidence, timing, and censoring. A person-level classifier, a sequence-level forecast, and an institution-level risk score can appear mathematically similar while providing different information. Steyerberg et al. (2010) sets out evaluation principles for prediction models whose outcomes, assessment times, and intended decisions must be explicitly defined.
Binary labels, continuous ratings, ordered responses, time-to-event outcomes, and repeated within-person deviations require appropriate loss functions and statistical designs. In depression-related social media research, De Choudhury et al. (2013) used retrospective self-reports of diagnosis and preceding posts to explore signatures relevant to onset; this is an instructive research demonstration, not a verified community screening program. Temporal ordering, retrospective recall, participant selection, and platform-specific behaviors shape what a reported model can establish. A prospective forecast should preserve a strict separation between inputs available at prediction time and later information that defines the outcome.
Prediction goals also differ in the burden imposed by errors. A research model estimating a personality dimension for aggregate analysis may tolerate types of error that would be unacceptable if the same output were used to deny assistance or assign a diagnosis. An operational threshold is a decision policy built on top of estimated probabilities. Van Calster et al. (2019) explains why calibrated probabilities matter for use, while Steyerberg et al. (2010) separates discrimination, calibration, and decision-analytic evaluation. The appropriate outcome, timing, threshold, comparison, and utility function should be specified before tuning algorithms to improve a headline accuracy metric.
Research Designs and the Hierarchy of Predictive Claims
A retrospective training experiment asks whether an algorithm can reconstruct or estimate recorded outcomes from available data. A temporally held-out evaluation asks whether the model transfers to later observations; an external-site assessment asks whether it works in a new setting; prospective implementation research asks whether the performance and workflow persist when inputs are collected in ordinary practice. These designs answer increasingly consequential questions. Chekroud et al. (2024) found strong within-trial prediction yet severe loss of generalization across separate clinical trials in the context investigated. Such evidence demonstrates why apparently excellent internal performance cannot be equated with reliable deployment in another clinical population.
Data-generating processes should be mapped before model fitting. Participants are nested in families, schools, clinics, workplaces, and platforms; individuals contribute repeated sessions, messages, or device measurements. Separating records at random can place highly related observations on both sides of a train–test divide. If stable person-specific cues are then used to predict later records from the same participant, the reported accuracy answers a different question from generalization to unseen people. Vabalas et al. (2019) examines optimistic evaluation in small-sample high-dimensional settings, while Riley et al. (2019) formalizes sample-size planning requirements for developing prediction models with binary outcomes.
A strong evidentiary sequence begins with a clearly stated target and sampling frame, constructs a locked preprocessing and modeling pipeline, reserves independent evaluation data, evaluates relevant baselines, and replicates results in changed conditions. In clinically significant applications, model studies can additionally be appraised with Wolff et al. (2019) and reported using Collins et al. (2024), which extends transparent reporting guidance to regression and machine-learning prediction. A reproducible script and an attractive performance plot establish a computational result; the broader scientific claim requires confidence that the evaluation reflects the people, outcome definitions, and institutional conditions for which the prediction is asserted.
Behavioral, Linguistic, and Multimodal Predictors
Predictor variables range from carefully scored test items to unstructured language, social interaction graphs, activity records, passive device measurements, and audiovisual signals. Their apparent richness conceals considerable dependence on data collection rules. Recorded location may vary with device permissions, transport infrastructure, occupational schedule, and socioeconomic resources; expressive language varies with audience, genre, and platform norms. Onnela & Rauch (2016) describes the scientific possibilities of smartphone-based behavioral observation, while Schwartz et al. (2013) demonstrates that open-vocabulary language analysis can reveal systematic associations with personality and demographic variables. Neither example makes a feature psychologically meaningful independently of its origin and context.
Before analysis, investigators must specify the observation window, missingness rules, aggregation interval, consent conditions, and unit represented by each feature. A phone carrying two people, a device used intermittently, or a social account curated for work complicates straightforward attribution of behavior to one person's stable characteristics. Multimodal fusion adds the further problem that speech, movement, and digital traces may be jointly influenced by surroundings rather than a shared latent psychological trait. Features should be audited for technical reliability, time stamps, repeated-person structure, and whether model developers have privileged access to future information. The empirical framework in Stachl et al. (2020) is valuable precisely because naturally observed smartphone data were compared with declared trait targets.
Preprocessing is an estimated component of the model. Vocabulary selection, imputation, scaling, dimensionality reduction, feature pruning, and representation learning must occur inside each training fold when the intended inference is about performance on new observations. If the entire dataset is standardized or screened using outcome associations before held-out testing, the test observations have already influenced the constructed predictor. This form of leakage can yield convincing but invalid estimates. Hastie et al. (2009) describes the dependence of fitted statistical rules on the development sample; Vabalas et al. (2019) documents why evaluation practices are especially delicate when many candidate features are available relative to the number of independent participants.
Model Families and Statistical Learning Principles
Simple linear and logistic regression are important scientific benchmarks because their coefficients, regularization choices, and prediction functions can be inspected and tested. Penalized generalized linear models address correlated predictors and high-dimensional vocabularies; tree ensembles represent interactions and nonlinearities; kernel methods learn flexible separation functions; neural networks can learn representations of language, sound, images, and sequential activity. Hastie et al. (2009) provides a unifying statistical treatment of these families. The model class should follow the data structure, available sample, interpretability needs, and target outcome rather than the anticipated prestige of a particular architecture.
Complexity produces a bias–variance problem and can increase sensitivity to idiosyncrasies of one cohort. Regularization, shrinkage, early stopping, and prespecified parameter search can reduce overfitting but do not repair a misspecified outcome or a nonrepresentative sample. Large language embeddings may reveal stable associations, yet if the language distribution changes across educational groups or platforms their apparent predictive advantage may diminish. Park et al. (2015) demonstrates that linguistic features can be evaluated against several personality criteria, while Chekroud et al. (2024) documents how promising clinical predictions may fail across trials even when an elaborate modeling pipeline is used. Predictive success therefore has to be re-established at the relevant transport boundary.
Comparison requires the same outcome, evaluation population, loss function, and access to information. A trivial prevalence-only rule, previously observed score, or conventional psychometric scale may be the relevant starting point. Where a model predicts current distress, a validated short questionnaire may be a stronger comparator than an elaborate machine classifier, depending on the application. Statistical significance of a difference in cross-validated scores is insufficient by itself to establish better decisions. Steyerberg et al. (2010) recommends comprehensive evaluation beyond a single discrimination statistic, and Yarkoni & Westfall (2017) emphasizes transparent out-of-sample performance against well-chosen alternative accounts.
Validation Partitions, Leakage, and Reproducible Pipelines
Cross-validation estimates how a complete fitted procedure may behave under a particular resampling scheme. It does not automatically simulate an independent future population. If a dataset contains repeated observations, separate participants rather than isolated messages when the scientific question concerns unseen persons. For organizational deployment, reserve entire institutions; for prospective forecasting, hold out later chronological periods. Nested resampling separates hyperparameter selection from outer performance estimation. Vabalas et al. (2019) shows that limited sample sizes and inadequate validation can inflate estimates, making explicit resampling design a substantive scientific requirement rather than a matter of software preference.
Leakage can originate in duplicates, copied text, hidden participant identifiers, labels encoded in feature names, post-outcome services, preprocessing applied before partitioning, or human raters using the same data that later appear as predictors. A language model exposed during pretraining to benchmark-like records can undermine a supposed held-out evaluation even when supervised training data were split correctly. The relevant criterion is informational independence: would the predictor have had access to this information at the moment when the target forecast was meant to be issued? Collins et al. (2024) stresses transparent reporting of input handling, modeling, validation, and prediction model performance, providing a reporting structure for documenting such choices.
Complete reproducibility includes the dataset version, feature definitions, eligibility criteria, split assignment, preprocessing code, random seeds, tuning budget, model coefficients or weights, and decision thresholds. Privacy or licensing may prevent public release of sensitive psychological records, so reproducibility may require controlled access, synthetic test fixtures, detailed data dictionaries, or an independent privacy-preserving evaluation team. Wolff et al. (2019) identifies recurrent domains of risk of bias in prediction-model research, including participants, predictors, outcomes, and analysis. Reporting a single final test accuracy without documenting the route by which the model was selected conceals multiple opportunities for optimistic estimation.
Discrimination, Calibration, and Predictive Uncertainty
Discrimination asks whether a model ranks observations meaningfully; calibration asks whether its numeric predictions correspond to observed outcome frequencies; overall performance combines aspects of accuracy; decision utility asks whether using the model improves actions under a specified trade-off between errors. For binary risk models, area under the receiver operating characteristic curve measures ranking ability without choosing a threshold. Precision and recall become especially relevant when outcomes are rare and false alarms impose meaningful costs. Steyerberg et al. (2010) offers a framework linking discrimination, Brier score, calibration, and decision analysis. Selecting one statistic after looking at the results can disguise a substantial failure of probability interpretation.
A model can discriminate relatively well while assigning badly calibrated probabilities. A reported risk of 0.8 should be examined among cases receiving predictions near that value, within a stated population and observation horizon. Calibration plots, intercepts, slopes, uncertainty intervals, and subgroup checks should be evaluated on independent records. Van Calster et al. (2019) argues that calibration has been systematically underappreciated relative to discrimination. A threshold optimized in a development sample may be unsuitable when outcome prevalence, available resources, or institutional costs change. Reliable decision support depends on explicit uncertainty and local clinical or organizational assessment rather than score ranking alone.
Performance estimates themselves vary with sampling. Confidence intervals should account for independent sampling units; ordinary record-level bootstrapping can understate uncertainty in datasets with repeated observations or clustered participants. For high-dimensional models, apparent gains should be assessed against the amount of tuning and the effective number of outcome events. Riley et al. (2019) proposes sample-size criteria for binary-outcome prediction development that include overfitting and precision of overall risk estimates. None of these methodological safeguards establishes the validity of the psychological construct: even a well-calibrated forecast can concern a weak, biased, or institutionally contingent outcome definition.
Sample Size, Missing Data, and Statistical Robustness
The effective sample size in digital psychology is the number of relevant independent units rather than simply the number of extracted words, movements, recordings, or clicks. A study with millions of messages from a modest number of volunteers cannot claim the same person-level representativeness as a cohort containing millions of independent participants. The events-per-parameter heuristic also becomes inadequate when feature selection and nonlinearity are extensive. Riley et al. (2019) develops principled sample-size considerations for clinical prediction with binary or event-time targets; Vabalas et al. (2019) provides empirical evidence that very small samples can yield misleadingly optimistic machine-learning evaluations.
Missing predictors carry psychological and social information. Withdrawal from an app may correlate with changing health, loss of connectivity, work demands, privacy preference, or technical failures; ignoring those differences can introduce selection bias. Imputation and missingness indicators can improve predictive continuity but do not recover systematically absent participants. Investigators should examine completeness by group and time, record consent-related exclusions, and test whether the deployment environment reproduces the same missingness pattern. Onnela & Rauch (2016) anticipated opportunities and burdens in continuously sampled behavioral information. A robust model evaluation measures degradation when devices, users, or communication channels change.
Stress tests can vary predictor windows, thresholds, sampling frames, feature sets, demographic stratifications, and calibration procedures. Predeclared sensitivity analysis guards against selectively presenting the best analysis. Yet robustness to selected analytic alternatives is only one component of validity: multiple pipelines can agree because they share the same biased labels or selected population. Flake & Fried (2020) explains how undocumented measurement choices threaten cumulative inference, while Wolff et al. (2019) supplies structured domains for evaluating prediction-model studies. An evidence-based report should state which analyses were planned, which were exploratory, and which uncertainties remain after sensitivity testing.
Digital Footprints and the Prediction of Personal Attributes
The study by Kosinski et al. (2013) demonstrated that digital records of Facebook Likes could predict several demographic and psychological attributes within a volunteer cohort. The authors combined dimensionality reduction with regression methods and used test-based, self-reported, and demographic targets. Their work established an influential proof of concept: apparently mundane traces of online behavior encode statistical information about personal characteristics. The same findings also imply a privacy problem because sensitive attributes may become inferable from data collected for unrelated purposes. Sampling through a particular platform, historical patterns of use, and self-selected participation restrict the population to which accuracy can be attributed.
In Youyou et al. (2015), computer-based personality assessments derived from Facebook Likes were evaluated against judgments provided by human acquaintances and questionnaire-based trait criteria. The reported comparisons showed that machine judgments could perform favorably relative to certain human assessors under the study's specific information and criterion conditions. The result is evidence for the utility of data-driven prediction of self-reported traits; it does not imply that automated systems possess direct psychological insight, outperform every trained professional, or produce equally valid assessments in all populations. The empirical question remains how information availability, criterion reliability, and comparison conditions shape the observed advantage.
The combined digital-footprint evidence shifts attention from whether prediction is possible to what is being predicted and how portable the relationship is. A behavior that strongly discriminates groups during one period can acquire a different social meaning after changes in platform demographics, privacy controls, or public norms. Digital traces also permit unintended sensitive-attribute inference even when the intended outcome is innocuous. Kosinski et al. (2013) and Youyou et al. (2015) therefore supply historical demonstrations of predictive association together with enduring questions about consent, feature meaning, sampling, and downstream use. A model should be understood as learning statistical patterns of recorded activity, with psychological interpretation requiring additional validation.
Natural Language as a Psychological Measurement Surface
Language is a distinctive predictor because it serves simultaneously as an action, a social performance, and a record of context. Schwartz et al. (2013) used an open-vocabulary approach to associate linguistic content with personality, age, and gender in a large social-media dataset. The work showed that category-free exploration can identify patterns excluded by researcher-selected dictionaries, supporting discovery of candidate linguistic markers. Its observational design also means that topics, audience, role, and life circumstances may contribute to prediction. A word associated with a trait in one platform community cannot be treated as a timeless marker of the same psychological attribute.
The more directly assessment-oriented investigation by Park et al. (2015) used language from Facebook users and compared trait predictions with self-report scales, informant ratings, test–retest observations, and external correlates. This multi-criterion strategy strengthens the assessment argument because it asks whether the predictions relate to several forms of evidence rather than one optimized label alone. The sample and linguistic environment still matter. A model trained on voluntary social-media writing may handle dictated speech, workplace messages, or texts produced under stress differently; the apparent similarity of words does not establish equivalence of the underlying response process.
Contemporary language models can transform documents into numerical representations that improve prediction for some target datasets, yet the representational basis remains dependent on pretraining distributions and task-specific fine-tuning. For mental-health contexts, linguistic cues can include self-description, discussion of treatment, and responses to prompts; these may be correlated with recorded diagnoses because the label is disclosed in the text. Detecting such cues is analytically different from detecting an unobserved psychological condition. De Choudhury et al. (2013) illustrates the need to reason about timing and reported clinical labels. Flake & Fried (2020) reinforces the requirement to describe precisely which behavior is measured and what interpretation the resulting score supports.
Smartphone Signals, Digital Phenotyping, and Ecological Observation
Smartphones generate longitudinal records of physical activity, communication rhythms, screen interactions, mobility, and context. Onnela & Rauch (2016) characterized the potential of these records for behavioral and mental-health research, including observation outside conventional laboratory encounters. Their principal value is ecological reach: repeated measurements may reveal within-person changes that brief retrospective questionnaires miss. Ecological proximity does not make an indicator unambiguous. Reduced mobility can follow illness, work schedules, weather, local transport restrictions, or personal preference. Any inference about mood or motivation from movement needs tests against alternative causal and contextual explanations.
In Stachl et al. (2020), patterns of observed smartphone behavior were used to predict personality traits. The investigation linked recorded usage features to established trait measurements and helped demonstrate the feasibility of computational inference from ordinary digital behavior. Its validity derives from the observed statistical relationships in the investigated sample and evaluation design. A device-based personality estimate is not inherently equivalent to a psychometric inventory, an informant assessment, or an individual's enduring psychological organization. Population-specific habits, device availability, operating-system restrictions, and changes in recording software may alter the same feature's interpretation.
Longitudinal device datasets introduce repeated-measure dependence and informative gaps. A feature describing one individual over many days can support personalized deviation detection, while cross-person trait estimation requires models validated on individuals absent from development. These targets should never share an evaluation design merely because they both use daily records. Monitoring may also affect behavior when people learn that their devices are being used for assessment. Stachl et al. (2020) and Onnela & Rauch (2016) support the scientific promise of behavioral sensing, while the methodological safeguards from Vabalas et al. (2019) indicate why independent participants, careful missing-data analysis, and prospective external testing are essential before claims of routine assessment accuracy.
Mental Health Prediction and the Limits of Clinical Transfer
Research on mental-health prediction ranges from symptom-score estimation to prediction of clinical status, future treatment response, relapse, or service contact. De Choudhury et al. (2013) examined associations between social-media behavior and self-reported depression, including observations preceding a reported episode. This was an influential early study of linguistic and social indicators, yet its collected labels and participant recruitment determine the scope of the findings. Social-media-derived predictions cannot be equated with a validated diagnostic interview or licensed clinical decision procedure. In clinical contexts, the consequences of false-positive and false-negative outputs require separate evaluation.
The review by Dwyer et al. (2018) surveyed the potential of machine-learning approaches in clinical psychology and psychiatry, while the scoping review by Shatte et al. (2019) examined applications and methods across mental-health research. Such reviews establish the breadth of investigation and expose recurrent obstacles around heterogeneity, variable validation, and translation. They do not supply a pooled performance estimate for every condition or model class. Different diagnostic definitions, assessment schedules, and study populations create fundamentally different prediction targets. Combining findings without accounting for these differences would provide a misleading impression of one homogeneous technology.
A sharp test of clinical transport appeared in Chekroud et al. (2024), which evaluated prediction of treatment outcomes across several schizophrenia clinical trials. Within-trial model success did not transfer successfully to held-out trials under the reported designs. This result directly challenges claims that cross-validation within one clinical dataset is sufficient evidence of usefulness in a new care setting. Kelly et al. (2019) describes additional barriers between development and impact, including workflow fit and real-world evaluation. The defensible application pathway includes external validation, local calibration, prospective testing, patient-safety review, and qualified clinical oversight; the current monograph offers research evaluation rather than individual diagnosis or treatment instructions.
Institutional Prediction, Selection, and Decision Consequences
Algorithmic assessment can influence employee selection, educational guidance, resource allocation, and psychological services. The relevant output may be an aptitude score, predicted performance, a behavioral-risk estimate, or a proposed level of intervention. In each setting, practical utility depends on the criterion and who controls the subsequent decision. Meehl (1954), Dawes et al. (1989), and Grove et al. (2000) illuminate the history of formal prediction compared with professional judgment, while Steyerberg et al. (2010) clarifies why a model's ranking performance alone does not show that a decision threshold benefits those affected. Each institution must establish its own relevant comparison and consequences.
Workplace or educational targets often reflect opportunities, evaluation procedures, and resource differences as much as individual capacities. Training a model on historical performance ratings can reproduce prior discrimination or managerial preference, even if the rating is statistically predictable. The detailed health-resource investigation by Obermeyer et al. (2019) reveals a related failure mechanism: using financial expenditure as a proxy for need produced consequential disparities. This analogy concerns target design and decision consequences rather than an unsupported assertion that all psychological prediction systems reproduce that particular bias. Evaluating fairness requires examining what the label rewards or penalizes and whether affected groups have comparable opportunities to produce it.
Decision effects should be assessed against current practice and feasible alternatives, including decisions without an algorithm. A model that improves one summary metric can increase unnecessary referrals or systematically miss a subgroup; benefits may also disappear if staff disregard alerts or change behavior in response to scores. Kelly et al. (2019) discusses barriers to clinical integration, and WHO (2021) identifies organizational accountability and human rights as core considerations for AI in health. The scientific claim of useful assessment should ultimately be tied to verified changes in outcomes or decisions under a defined governance arrangement, not merely to an impressive development-dataset correlation.
Transportability Across Populations and Institutions
Generalization is a relation between a learned prediction function and a new data-generating environment. Differences in age, language, culture, educational opportunity, treatment protocol, recruitment, measurement platform, and outcome prevalence may alter its performance. An internal random hold-out mainly tests how the model behaves under a related sampling regime. A geographically or institutionally external test asks a different question: does the learned relation survive a substantive environmental change? Chekroud et al. (2024) provides a striking example of models trained in one psychiatric trial losing predictive success elsewhere. External validation must name the change being tested, and representativeness must be evaluated against the intended deployment population.
Statistical transport may fail through changing predictor distributions, changed relationships between predictors and outcomes, or altered outcome definitions. These mechanisms call for different responses. Updating the calibration intercept may help when baseline outcome rates shift but predictor effects remain stable; recalibration alone cannot repair a label that has a different meaning in another clinic or school. Van Calster et al. (2019) discusses the fundamental importance of calibration, while Wolff et al. (2019) highlights applicability in evaluating model evidence. Researchers should examine whether data collected under a particular research protocol can be reproduced in ordinary settings, where missingness, workflow constraints, and consent patterns differ.
Transportability assessment should make uncertainty visible at the subgroup and site levels, with enough observations to estimate performance rather than merely display unstable small-group plots. Prevalence differences also change predictive values even if ranking characteristics remain similar. Claims about nationwide use cannot be derived from a volunteer platform cohort without independent evidence about sampling and conditional relationships. Park et al. (2015) and Stachl et al. (2020) support domain-specific feasibility; Chekroud et al. (2024) shows why success is not an automatic guarantee of clinical transfer. The appropriate synthesis is conditional: each model must establish which features, labels, and outcomes remain stable at its intended boundary of use.
Fairness, Differential Error, and the Choice of Ground Truth
Algorithmic fairness begins with the quality of the target and the population it represents. Comparing errors across demographic groups is necessary but may be insufficient when the target itself embeds unequal access, recognition, or institutional treatment. Obermeyer et al. (2019) demonstrated this mechanism in an algorithm used to identify patients with health needs: expenditure was an inadequate proxy for health burden because patterns of spending differed across groups. Psychological assessments face analogous possibilities when school grades, hiring recommendations, disciplinary records, or clinical access are treated as uncomplicated measures of ability, risk, or well-being. Ground-truth review must therefore precede subgroup performance dashboards.
Fairness criteria can conflict mathematically when base rates and error distributions differ. Equalizing false-negative rates, achieving calibration within each group, and imposing identical decision thresholds may lead to distinct allocations; no single aggregate fairness statistic resolves the normative question of which errors are most consequential. A transparent assessment must specify the intended benefit, the groups at risk, and the consequences of mistaken decisions. AERA, APA, & NCME (2014) addresses the responsibilities involved in test fairness and interpretation; Tabassi (2023) provides broader guidance for managing AI risks. The choice of equity criterion should be documented and justified in the actual use context.
Intersectional and contextual analysis matters because a model may appear similar across broad categories while exhibiting systematic failures for people who differ by language, disability, age, or access to digital technology. Demographic prediction from language or Likes, documented in Schwartz et al. (2013) and Kosinski et al. (2013), does not by itself justify using those inferences to determine access to services or opportunities. Data minimization, legitimate purpose, meaningful review, and opportunities to correct consequential errors are part of responsible implementation. The empirical task is to test differential errors on genuinely independent observations; the governance task is to decide whether a particular inference should be made or used at all.
Interpretability, Explanations, and Mechanistic Claims
An algorithm's reported predictors are statistical features; they need not represent causal psychological mechanisms. A model assigning weight to a linguistic expression may be exploiting topic, age, platform membership, or opportunity structure rather than the cognitive process attributed to that expression. Post-hoc feature importance describes aspects of a fitted mapping under a chosen explanatory procedure; it cannot independently prove the meaning or causal role of the feature. Rudin (2019) argues for inherently interpretable models in high-stakes settings where transparency is itself essential. This provides a methodological direction: begin with understandable baselines, establish whether additional complexity yields a meaningful and independently validated benefit, and scrutinize the limits of explanations.
Mechanistic explanations require experiments, intervention evidence, or strong causal assumptions beyond standard outcome prediction. Yarkoni & Westfall (2017) encourages psychology to take predictive performance seriously, but this methodological argument does not erase the distinction between explanation and forecasting. A feature can improve generalization without being a cause; conversely, a genuine causal variable may yield weak prediction when individual outcomes are noisy or additional causes remain unmeasured. Cronbach & Meehl (1955) emphasizes relations among theoretical constructs and observations, reminding researchers that a computational score should be integrated into a tested interpretive network rather than used as an unexamined substitute for psychological theory.
Interpretability should be tested with the people who must understand and contest decisions. An elegant coefficient table can still mislead when the measured target is defective, and a visually compelling heatmap can obscure unstable correlations. Model cards, feature descriptions, documented thresholds, and plain-language uncertainty statements improve accountability only when they are accurate and comprehensible to users. Collins et al. (2024) outlines transparent reporting for prediction studies; WHO (2021) addresses governance obligations in health applications. Human oversight must have an operational meaning: qualified reviewers need appropriate information, authority to revise decisions, and procedures for monitoring harm instead of merely being nominally present.
Deployment Evaluation, Monitoring, and Model Drift
Model performance can change after installation even when the program remains technically functional. Platform updates alter available signals; users modify behavior when they understand the assessment; institutions change eligibility criteria; and diagnoses or rating conventions evolve. Monitoring therefore examines data quality, prediction distributions, observed outcomes, calibration, subgroup performance, and the resulting decisions over time. Kelly et al. (2019) describes the organizational and scientific difficulties of producing real clinical benefit from AI, while Tabassi (2023) provides a lifecycle approach to managing AI risk. A single successful validation study is a baseline for monitored use, not a perpetual certificate.
Prospective implementation studies should evaluate both model performance and decision consequences. If a risk score prompts additional screening, the screening process may change measured outcomes and produce feedback loops. If only high-scoring people receive detailed assessments, future labels become selectively observed. This complicates apparent performance improvement and requires designed follow-up in all relevant groups where ethically and operationally feasible. Steyerberg et al. (2010) emphasizes decision-analytic measures; WHO (2021) stresses accountability and safe governance. Appropriate study designs may include staged pilots, independent safety review, comparison with current workflows, and evaluation of how human professionals use the results.
Updating a model should trigger renewed evaluation, documentation, and version control. Replacing a feature extractor, changing the label source, or retraining on outcomes affected by prior model decisions modifies the prediction system's meaning even if the webpage still displays the same name. Investigators should distinguish data drift from concept drift, and separate recalibration from broader model redevelopment. Collins et al. (2024) describes essential reporting features for prediction models, while Wolff et al. (2019) structures bias and applicability assessment. The continuing scientific question is whether the model serves the intended population and decision under contemporary conditions, with evidence sufficient to justify continued use or timely withdrawal.
Privacy, Consent, and Governance of Psychological Inference
Psychological data can be sensitive because their combination permits inference beyond what individuals knowingly supplied. Kosinski et al. (2013) showed that volunteered Likes could statistically reveal highly sensitive attributes; the research implications persist for inferred identity, beliefs, and vulnerabilities. The principle of purpose limitation is central: permission to store information for social interaction is not necessarily permission to profile psychological characteristics for recruitment, insurance, or care decisions. Assessment proposals should identify a legitimate scientific or institutional purpose, define who may access raw signals and inferred labels, and explain retention, reuse, and deletion practices.
Consent needs to reflect the actual data pipeline and foreseeable consequences. Passive sensing can collect observations about locations, contacts, schedules, and bystanders who did not directly participate. Onnela & Rauch (2016) outlines the opportunities for digital phenotyping, making careful governance of such data essential. Secure computation and access controls reduce some exposure but cannot correct invalid labels or unjust decisions. Collection minimization, aggregation, audit logs, independent ethics review, and controlled access to sensitive research datasets are compatible with reproducible scientific practice. Publishing a model's code does not require publicly exposing participants' private records.
Governance frameworks should distinguish scientific feasibility from authorized deployment. WHO (2021) articulates principles for ethical management of health-related AI, and Tabassi (2023) organizes risk governance, mapping, measurement, and management for broader AI systems. Psychological applications also remain accountable to professional testing standards AERA, APA, & NCME (2014) and any applicable clinical regulations. The evidence presented here supports research on transparent risk and model evaluation. It cannot establish a blanket legal status for any platform or jurisdiction, and it supplies no individualized clinical advice. Ethical quality is determined in part by how errors, appeals, privacy obligations, and oversight are handled in the actual institution.
Prediction, Psychometric Evidence, and Causal Explanation
A predictive function and a psychological explanation are different scientific achievements. Prediction establishes a conditional relationship between observed inputs and a target under specified sampling conditions. Psychometric validation asks whether the measured score warrants an interpretation about an intended construct. Causal explanation asks what would change under a relevant intervention or counterfactual condition. Yarkoni & Westfall (2017) explains the value of predictive evaluation to psychological science; Cronbach & Meehl (1955) makes explicit the wider theoretical and evidentiary problem of construct interpretation. These tasks reinforce one another when their claims and designs are clearly articulated, but no one task automatically settles the others.
The multi-method perspective provides a useful bridge. Campbell & Fiske (1959) proposed comparing convergent and discriminant patterns across traits and methods as evidence about measurement; Park et al. (2015) put related concerns into an algorithmic personality-assessment setting by assessing language-derived traits against several external criteria. In both cases, cross-method agreement improves a validity argument only to the extent that methods do not share the same source of error or selection. A prediction that reproduces a self-report scale can remain biased by the scale's wording, its cultural assumptions, or the recruitment procedure. Model developers should therefore distinguish prediction of one measure from convergent evidence across genuinely independent indicators.
An intervention may also alter the predictor–outcome relation. A device-use pattern associated with mood under observation need not remain associated after a therapeutic program, workplace change, or platform intervention. Studying whether acting on a score improves outcomes requires new evidence rather than an inference from retrospective accuracy. Steyerberg et al. (2010) distinguishes predictive measures from decision utility; Chekroud et al. (2024) shows the difficulty of transporting even a well-performing prediction into changed clinical conditions. A mature psychological assessment program integrates conceptual validity, independent performance, causal uncertainty, and the practical consequences of score use.
Human–Artificial Assessment and the Boundaries of Attribution
The Artificial Era creates new settings in which computational systems evaluate human behavior, collaborate with human professionals, and themselves become subjects of behavioral testing. Psychological prediction of humans through algorithms concerns a clearly identified human target and observed human data. Evaluation of an artificial system's performance concerns different entities, tasks, and measurement assumptions. Numerical similarities between human and artificial test scores do not establish equivalence of mental organization. The general rules of construct validity from Cronbach & Meehl (1955) and the caution about algorithmic explanation from Rudin (2019) help maintain disciplined attribution across these research settings.
When language-generating systems are used as assessors, one must test stability across prompts, versions, dialects, contexts, task instructions, and protected characteristics. The generated judgment is a system output subject to its own validation and monitoring, not evidence of privileged access to a person's internal experience. When an artificial agent is the object assessed, performance should be described through observable behavior, reproducible task conditions, and operationally defined functional capacities. Claims about intelligence, identity, agency, or psychological continuity require separate evidence. The methodological principles in AERA, APA, & NCME (2014), Collins et al. (2024), and Tabassi (2023) offer applicable safeguards at the level of interpretation, reporting, and deployment risk.
Human–artificial partnerships introduce a feedback problem: professionals may change their own judgments after seeing an automated score, while systems may learn from decisions already affected by earlier outputs. Scientific evaluation should test the resulting joint decision process rather than assume that adding an algorithm improves professional accuracy. Dawes et al. (1989) and Grove et al. (2000) remain useful historical comparisons of formal and judgmental combination, while Kelly et al. (2019) highlights the difficulties of delivering real value through technology. This interdisciplinary extension remains grounded in observable interactions and empirical tests; philosophical interpretations within the Artificial Era constitute a distinct level of analysis.
An Evidence-Based Agenda for Future Research
Future research should establish comparable external validation designs across genuinely independent locations, populations, languages, and technology environments. The striking cross-trial finding of Chekroud et al. (2024) suggests that transport assessment belongs near the beginning of the research program, especially for clinical applications. Researchers should report which predictor definitions remain available at new sites, whether outcome criteria are equivalent, and how calibration and subgroup errors change. A successful multi-site result would be stronger evidence for practical generalization than repeated optimization within a single institutional dataset. Shared evaluation protocols can facilitate this work without demanding uncontrolled release of sensitive data.
Measurement research should develop independent and sometimes adversarial criterion assessments that examine whether computational predictions converge with psychological constructs for the right reasons. Longitudinal evidence may distinguish stable person characteristics from temporary state, situation, language genre, or institutional exposure. Campbell & Fiske (1959) motivates cross-method validation; Stachl et al. (2020) illustrates naturalistic behavioral measurement; Flake & Fried (2020) identifies the importance of transparent operational choices. Studies should predefine what would count as a failure of construct interpretation, examine response processes where feasible, and preserve the distinction between latent constructs and observable administrative targets.
Implementation research should evaluate human decisions, false alarms, missed cases, appeal pathways, and outcomes after models influence practice. In health settings, WHO (2021), Collins et al. (2024), and Wolff et al. (2019) help structure governance, reporting, and risk-of-bias appraisal. Comparative experiments could test whether interpretable scores improve professional decisions relative to standard practice, with careful monitoring for automation bias and unequal burden. Relevant studies also need sufficient sample size, independent auditing, and follow-up periods appropriate to the decision. Scientific progress will depend on reliable evaluation of particular prediction systems, rather than unqualified claims that an entire class of algorithms is accurate or fair.
Methodological Boundaries and Appraisal of the Evidence
This monograph is a theory-guided critical narrative synthesis. It integrates selected foundational texts, original digital-assessment studies, predictive-methodology articles, clinical studies, reviews, and professional governance documents. It does not claim a comprehensive search protocol, risk-of-bias scoring for every possible eligible article, pooled meta-analytic estimates, or newly derived effect sizes. These limits matter: published examples emphasize notable successes and failures, while unpublished systems, proprietary models, and platform-specific data may remain inaccessible. The review by Shatte et al. (2019) documents the broad and heterogeneous mental-health machine-learning literature; Dwyer et al. (2018) provides its earlier clinical research context.
The evidence has different strengths for different claims. Digital-language and smartphone studies such as Park et al. (2015) and Stachl et al. (2020) provide direct observational tests for selected trait targets. Youyou et al. (2015) evaluates particular human-versus-computational judgment comparisons. Chekroud et al. (2024) supplies an unusually informative independent test of clinical transport, but its specific schizophrenia treatment outcomes are not a numerical estimate for all psychological models. Obermeyer et al. (2019) documents consequential proxy-target bias in healthcare and establishes a mechanistic caution relevant to psychological applications, without measuring bias in every psychological assessment system. Wolff et al. (2019) and Collins et al. (2024) provide appraisal and reporting frameworks rather than direct demonstrations of clinical benefit.
The strongest cross-cutting conclusion is methodological. Prediction requires a prespecified outcome, independently evaluated performance, and appropriate uncertainty; assessment additionally requires a defensible argument for interpreting the score as evidence about a psychological construct; applied use requires monitoring the decisions and consequences produced by that score. Steyerberg et al. (2010), Van Calster et al. (2019), Cronbach & Meehl (1955), and AERA, APA, & NCME (2014) jointly support this structure from distinct statistical and psychometric traditions. The main unresolved empirical questions concern population transport, long-term stability, latent construct validity, performance in rare or underrepresented subgroups, and the causal effects of institutional reliance on automated judgments.
Conclusion
Algorithmic psychological assessment and predictive modeling extend a century of formal judgment and psychometric investigation into environments of high-dimensional, continuous, and socially embedded data. The historical sequence from Meehl (1954) through Grove et al. (2000) to contemporary research indicates the durable value of reproducible rules and genuinely independent comparison. Studies of digital footprints, language, and smartphones demonstrate that behavior contains statistical signals relevant to psychological questionnaires and some outcomes Kosinski et al. (2013) Park et al. (2015) Stachl et al. (2020). The scientific interpretation of those signals remains dependent on the quality of the target, participant sampling, assessment context, and the stability of the learned relationship.
Evidence of predictive accuracy develops meaning through validation design. Carefully separated datasets, honest baselines, calibration, uncertainty analysis, examination of missingness, and independent tests of transport determine the credibility of performance claims. Vabalas et al. (2019) illustrates risks of optimistic small-sample evaluation; Chekroud et al. (2024) documents the failure of some clinical outcome predictions across trials; Obermeyer et al. (2019) demonstrates how the wrong target can produce consequential inequity. These findings establish the need to evaluate what is predicted and for whom before any practical decision is justified. Statistical association, construct interpretation, and clinical or institutional usefulness must each receive their appropriate empirical evidence.
A complete assessment system remains accountable after initial validation. Interpretability, privacy, fairness, access to review, versioning, monitoring, and impact evaluation connect computational prediction to the lives of the people represented by its scores. The methodological guidance of AERA, APA, & NCME (2014), Collins et al. (2024), WHO (2021), and Tabassi (2023) makes these responsibilities explicit across testing, clinical reporting, and AI governance. Within the wider transition From Homo to Artificial, the same discipline supports analysis of human–artificial decision systems through observable capacities, operational definitions, and reproducible evaluation. The enduring scientific task is precise prediction joined to valid psychological interpretation and demonstrable consequences.
References
- American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association. https://ncme.org/resources/books/testing-standards/
- Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait–multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016
- Chekroud, A. M., Hawrilenko, M., Loho, H., Bondar, J., Gueorguieva, R., Hasan, A., Kambeitz, J., Corlett, P. R., Koutsouleris, N., Krumholz, H. M., Krystal, J. H., & Paulus, M. (2024). Illusory generalizability of clinical prediction models. Science, 383(6679), 164–167. https://doi.org/10.1126/science.adg8538
- Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., et al. (2024). TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 385, e078378. https://doi.org/10.1136/bmj-2023-078378
- Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
- Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243(4899), 1668–1674. https://doi.org/10.1126/science.2648573
- De Choudhury, M., Gamon, M., Counts, S., & Horvitz, E. (2013). Predicting depression via social media. Proceedings of the International AAAI Conference on Web and Social Media, 7(1), 128–137. https://doi.org/10.1609/icwsm.v7i1.14432
- Dwyer, D. B., Falkai, P., & Koutsouleris, N. (2018). Machine learning approaches for clinical psychology and psychiatry. Annual Review of Clinical Psychology, 14, 91–118. https://doi.org/10.1146/annurev-clinpsy-032816-045037
- Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465. https://doi.org/10.1177/2515245920952393
- Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19–30. https://doi.org/10.1037/1040-3590.12.1.19
- Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7
- Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019). Key challenges for delivering clinical impact with artificial intelligence. BMC Medicine, 17, Article 195. https://doi.org/10.1186/s12916-019-1426-2
- Kosinski, M., Stillwell, D., & Graepel, T. (2013). Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, 110(15), 5802–5805. https://doi.org/10.1073/pnas.1218772110
- Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. University of Minnesota Press. https://meehl.umn.edu/lane-practice
- Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342
- Onnela, J.-P., & Rauch, S. L. (2016). Harnessing smartphone-based digital phenotyping to enhance behavioral and mental health. Neuropsychopharmacology, 41(7), 1691–1696. https://doi.org/10.1038/npp.2016.7
- Park, G., Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Kosinski, M., Stillwell, D. J., Ungar, L. H., & Seligman, M. E. P. (2015). Automatic personality assessment through social media language. Journal of Personality and Social Psychology, 108(6), 934–952. https://doi.org/10.1037/pspp0000020
- Riley, R. D., Snell, K. I. E., Ensor, J., Burke, D. L., Harrell, F. E., Jr., Moons, K. G. M., & Collins, G. S. (2019). Minimum sample size for developing a multivariable prediction model: Part II—Binary and time-to-event outcomes. Statistics in Medicine, 38(7), 1276–1296. https://doi.org/10.1002/sim.7992
- Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1, 206–215. https://doi.org/10.1038/s42256-019-0048-x
- Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D., Seligman, M. E. P., & Ungar, L. H. (2013). Personality, gender, and age in the language of social media: The open-vocabulary approach. PLOS ONE, 8(9), e73791. https://doi.org/10.1371/journal.pone.0073791
- Shatte, A. B. R., Hutchinson, D. M., & Teague, S. J. (2019). Machine learning in mental health: A scoping review of methods and applications. Psychological Medicine, 49(9), 1426–1448. https://doi.org/10.1017/S0033291719000151
- Stachl, C., Au, Q., Schoedel, R., Gosling, S. D., Harari, G. M., Buschek, D., Völkel, S. T., Schuwerk, T., Oldemeier, M., Ullmann, T., Hussmann, H., Bischl, B., & Bühner, M. (2020). Predicting personality from patterns of behavior collected with smartphones. Proceedings of the National Academy of Sciences, 117(30), 17680–17687. https://doi.org/10.1073/pnas.1920484117
- Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010). Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology, 21(1), 128–138. https://doi.org/10.1097/EDE.0b013e3181c30fb2
- Tabassi, E. (2023). Artificial intelligence risk management framework (AI RMF 1.0) (NIST AI 100-1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-1
- Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019). Machine learning algorithm validation with a limited sample size. PLOS ONE, 14(11), e0224365. https://doi.org/10.1371/journal.pone.0224365
- Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019). Calibration: The Achilles heel of predictive analytics. BMC Medicine, 17, Article 230. https://doi.org/10.1186/s12916-019-1466-7
- Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019). PROBAST: A tool to assess the risk of bias and applicability of prediction model studies. Annals of Internal Medicine, 170(1), 51–58. https://doi.org/10.7326/M18-1376
- World Health Organization. (2021). Ethics and governance of artificial intelligence for health: WHO guidance. World Health Organization. https://www.who.int/publications/i/item/9789240029200
- Yarkoni, T., & Westfall, J. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12(6), 1100–1122. https://doi.org/10.1177/1745691617693393
- Youyou, W., Kosinski, M., & Stillwell, D. (2015). Computer-based personality judgments are more accurate than those made by humans. Proceedings of the National Academy of Sciences, 112(4), 1036–1040. https://doi.org/10.1073/pnas.1418680112