Title |
A user-friendly tool to transform large scale administrative data into wide table format using a mapreduce program with a pig latin based script
|
---|---|
Published in |
BMC Medical Informatics and Decision Making, December 2012
|
DOI | 10.1186/1472-6947-12-151 |
Pubmed ID | |
Authors |
Hiromasa Horiguchi, Hideo Yasunaga, Hideki Hashimoto, Kazuhiko Ohe |
Abstract |
Secondary use of large scale administrative data is increasingly popular in health services and clinical research, where a user-friendly tool for data management is in great demand. MapReduce technology such as Hadoop is a promising tool for this purpose, though its use has been limited by the lack of user-friendly functions for transforming large scale data into wide table format, where each subject is represented by one row, for use in health services and clinical research. Since the original specification of Pig provides very few functions for column field management, we have developed a novel system called GroupFilterFormat to handle the definition of field and data content based on a Pig Latin script. We have also developed, as an open-source project, several user-defined functions to transform the table format using GroupFilterFormat and to deal with processing that considers date conditions. |
X Demographics
Geographical breakdown
Country | Count | As % |
---|---|---|
India | 1 | 50% |
United Kingdom | 1 | 50% |
Demographic breakdown
Type | Count | As % |
---|---|---|
Practitioners (doctors, other healthcare professionals) | 1 | 50% |
Members of the public | 1 | 50% |
Mendeley readers
Geographical breakdown
Country | Count | As % |
---|---|---|
Unknown | 38 | 100% |
Demographic breakdown
Readers by professional status | Count | As % |
---|---|---|
Student > Master | 9 | 24% |
Researcher | 7 | 18% |
Student > Bachelor | 5 | 13% |
Student > Ph. D. Student | 4 | 11% |
Student > Doctoral Student | 3 | 8% |
Other | 4 | 11% |
Unknown | 6 | 16% |
Readers by discipline | Count | As % |
---|---|---|
Computer Science | 11 | 29% |
Medicine and Dentistry | 8 | 21% |
Social Sciences | 3 | 8% |
Nursing and Health Professions | 2 | 5% |
Agricultural and Biological Sciences | 2 | 5% |
Other | 7 | 18% |
Unknown | 5 | 13% |