Publications

Abstract

The ongoing revolution in information technology is reshaping human life. In the realm of health behavior, wearable technology emerges as a leading digital solution, capturing physical behaviors (i.e., physical activity, sedentary habits, sleep patterns) within the 24-h cycle of daily life. Wearables are applied in research, clinical practice, and as lifestyle devices; most obvious, they promise to be a key element for increasing human physical activity, one of the biggest health challenges nowadays. However, despite the high expectations associated with wearable technology, fundamental aspects remain surprisingly neglected. Here, the lack of methodologically sound validation studies for wearables entering the market appears particularly critical. In our recent and comprehensive review of 967 validation studies, the overall low study quality was evident and alarming. In essence, standard regulatory processes are missing, although the scientific community is strongly advocating for improvements in the validation and trustworthiness of digital health products. Hence, we call for validity of wearables to be systematically tested to pave the way towards expedient digital health solutions and, in terms of reproducibility, to provide transparent information about the devices used (i.e., all data processing steps, analytical approaches, updates of algorithms). Toward this end, our viewpoint compiles challenges and suggests key elements for enhancing the quality of validation protocols (i.e., wearing position, criterion measurement, validated outcomes, sample size, statistical analyses) as well as issues pertinent to improving the validation process (i.e., replication of studies, access to raw data, the release of a new version). Moreover, to catalyze this comprehensive validation process, we are launching and introducing the project Wearable Landscape (www.wearable-landscape.info); the open science initiative not only compiles validation protocols but also facilitates collaboration efforts for sharing resources, equipment, set-up of multi-location studies, as well as joint data analysis and pre-processing. 

Body-worn devices that estimate physical behavior have tremendous potential to address key research gaps. However, there is no consensus on how devices and processing methods should be developed and evaluated, resulting in large differences in summary estimates and confusion for end users. We propose a phase-based framework for developing and evaluating devices that emphasizes robust validation studies in naturalistic conditions.

Key Points

• Body-worn devices have emerged as the measurement tool of choice in many surveillance, experimental, and observational studies.
• There has been a rapid and relentless proliferation of new devices and data processing methods without an established framework to evaluate their validity. This has resulted in much confusion, driven by nonrobust device development and evaluation methods that do not reflect how the devices will be used in practice.
• Divergence in summary estimates within and between devices obstructs efforts to pool data and preclude between-study comparisons, which are necessary to develop coherent public health recommendations.
• We propose the adoption of a phase-based framework to enable the field to move forward in a more systematic manner.
• This article seeks to stimulate discussion on such a framework by providing goals and recommendations for study methods at each phase and outlining key challenges facing the field.

Abstract

Consumer wearable and smartphone devices provide an accessible means to objectively measure physical activity (PA) through step counts. With the increasing proliferation of this technology, consumers, practitioners and researchers are interested in leveraging these devices as a means to track and facilitate PA behavioural change. However, while the acceptance of these devices is increasing, the validity of many consumer devices have not been rigorously and transparently evaluated. The Towards Intelligent Health and Well-Being Network of Physical Activity Assessment (INTERLIVE) is a joint European initiative of six universities and one industrial partner. The consortium was founded in 2019 and strives to develop best-practice recommendations for evaluating the validity of consumer wearables and smartphones. This expert statement presents a best-practice consumer wearable and smartphone step counter validation protocol. A two-step process was used to aggregate data and form a scientific foundation for the development of an optimal and feasible validation protocol: (1) a systematic literature review and (2) additional searches of the wider literature pertaining to factors that may introduce bias during the validation of these devices. The systematic literature review process identified 2897 potential articles, with 85 articles deemed eligible for the final dataset. From the synthesised data, we identified a set of six key domains to be considered during design and reporting of validation studies: target population, criterion measure, index measure, validation conditions, data processing and statistical analysis. Based on these six domains, a set of key variables of interest were identified and a ‘basic’ and ‘advanced’ multistage protocol for the validation of consumer wearable and smartphone step counters was developed. The INTERLIVE consortium recommends that the proposed protocol is used when considering the validation of any consumer wearable or smartphone step counter. Checklists have been provided to guide validation protocol development and reporting. The network also provide guidance for future research activities, highlighting the imminent need for the development of feasible alternative ‘gold-standard’ criterion measures for free-living validation. Adherence to these validation and reporting standards will help ensure methodological and reporting consistency, facilitating comparison between consumer devices. Ultimately, this will ensure that as these devices are integrated into standard medical care, consumers, practitioners, industry and researchers can use this technology safely and to its full potential.

Abstract

Background: Consumer wearables and smartphone devices commonly offer an estimate of energy expenditure (EE) to assist in the objective monitoring of physical activity to the general population. Alongside consumers, healthcare professionals and researchers are seeking to utilise these devices for the monitoring of training and improving human health. However, the methods of validation and reporting of EE estimation in these devices lacks rigour, negatively impacting on the ability to make comparisons between devices and provide transparent accuracy. Objectives: The Towards Intelligent Health and Well-Being Network of Physical Activity Assessment (INTERLIVE) is a joint European initiative of six universities and one industrial partner. The network was founded in 2019 and strives towards developing best-practice recommendations for evaluating the validity of consumer wearables and smartphones. This expert statement presents a best-practice validation protocol for consumer wearables and smartphones in the estimation of EE. Methods: The recommendations were developed through (1) a systematic literature review; (2) an unstructured review of the wider literature discussing the potential factors that may introduce bias during validation studies; and (3) evidence-informed expert opinions from members of the INTERLIVE network. Results: The systematic literature review process identified 1645 potential articles, of which 62 were deemed eligible for the final dataset. Based on these studies and the wider literature search, a validation framework is proposed encompassing six key domains for validation: the target population, criterion measure, index measure, testing conditions, data processing and the statistical analysis. Conclusions: The INTERLIVE network recommends that the proposed protocol, and checklists provided, are used to standardise the testing and reporting of the validation of any consumer wearable or smartphone device to estimate EE. This in turn will maximise the potential utility of these technologies for clinicians, researchers, consumers, and manufacturers/developers, while ensuring transparency, comparability, and replicability in validation.

Abstract 

Background: Consumer wearable technologies have become ubiquitous, with clinical and non-clinical populations leverag ing a variety of devices to quantify various aspects of health and wellness. However, the accuracy with which these devices measure biometric outcomes such as heart rate, sleep and physical activity remains unclear. Objective To conduct a ‘living’ (i.e. ongoing) evaluation of the accuracy of consumer wearable technologies in measuring various physiological outcomes. Methods: A systematic search of the literature was conducted in the following scientific databases: MEDLINE via PubMed, Embase, Cinahl and SPORTDiscus via EBSCO. The inclusion criteria required systematic reviews or meta-analyses that evaluated the validation of consumer wearable devices against accepted reference standards. In addition to publication details, review protocol, device specifics and a summary of the authors’ results, we extracted data on mean absolute percentage error (MAPE), pooled absolute bias, intraclass correlation coefficients (ICCs) and mean absolute differences. Results: Of 904 identified studies through the initial search, 24 systematic reviews met our inclusion criteria; these systematic reviews included 249 non-duplicate validation studies of consumer wearable devices involving 430,465 participants (43% female). Of the commercially available wearable devices released to date, approximately 11% have been validated for at least one biometric outcome. However, because a typical device can measure a multitude of biometric outcomes, the number of validation studies conducted represents just 3.5% of the total needed for a comprehensive evaluation of these devices. For heart rate, wearables showed a mean bias of ± 3%. In arrhythmia detection, wearables exhibited a pooled sensitivity and specificity of 100% and 95%, respectively. For aerobic capacity, wearables significantly overestimated VO2max by ± 15.24% during resting tests and ± 9.83% during exercise tests. Physical activity intensity measurements had a mean absolute error ranging from 29 to 80%, depending on the intensity of the activity being undertaken. Wearables mostly underestimated step counts (mean absolute percentage errors ranging from − 9 to 12%) and energy expenditure (mean bias = − 3 kcal per minute, or − 3%, with error ranging from− 21.27 to 14.76%). For blood oxygen saturation, wearables showed a mean absolute difference of up to 2.0%. Sleep measurement showed a tendency to overestimate total sleep time (mean absolute percentage error typically > 10%). Conclusions: While consumer wearables show promise in health monitoring, a conclusive assessment of their accuracy is impeded by pervasive heterogeneity in research outcomes and methodologies. There is a need for standardised validation protocols and collaborative industry partnerships to enhance the reliability and practical applicability of wearable technology assessments. Prospero ID: CRD42023402703.