Metrics, Measurement, and Online Visibility Digital Social Research: Methods Options - Group B Academic Year: 2015-16, Hilary Term Day and time: Weeks 6-9, Fridays 11:30-1:30 Location: OII Meeting Room, 1 St Giles Course Convenor Eric T. Meyer, Associate Professor, eric.meyer@oii.ox.ac.uk, Tel. 287218 Guest Lecturers Mike Thelwall, Professor, Statistical Cybermetrics Research Group, University of Wolverhampton Background Methods for measuring the impact of authors and publications have been used in certain fields (such as library and information science) since at least the 1960s. In the last decade, however, metrics and measurements of impact have both become easier to do (with most data available online), and more widespread (with the rapid growth of the importance of measures such as the h-index across all disciplines). While these metrics have been applied to scientific and academic outputs, similar techniques can also be used to measure economic activity, advertising attention, political visibility of parties and movements, the online prevalence of languages and cultural materials, and many other phenomena of interest to social scientists. Course Objectives and Outcomes This course is designed to give students experience analysing data both from traditional sources (Web of Knowledge, Scopus, etc.) and from alternative sources (Google, social media, Wikipedia, etc.). At the end of the course students will be able to: Describe the strengths and weaknesses of various digital approaches to measuring the impact of written materials Extract, clean, and analyse data from a variety of sources of metrics information Present metrics data in informative formats Teaching Arrangements The course will be taught during the second half of Hilary term in four weekly classes, consisting of a mix of lectures, hands-on work, student presentations, and seminar discussion. Formative Assessment Each student will be required to submit formative work throughout the term. Every week (except the last) all students will be required to submit a formative assignment that uses the techniques taught in that week s session. Note that the format of each assignment is similar, but each uses different data
sources and analytic techniques. These are each due the week after the class, and should be submitted by noon on the Wednesday following the class via Plato. These three assignments will contribute to the final summative report, as much of the data can be included in appendices to the final report. Summative Assessment The course will be formally assessed by means of a final summative report of approximately 2,500 words on the project each student has carried out during the course. This report will focus on a short but critical analysis of related literature, a presentation of findings, suggestions for future work, and several required appendices (which are in addition to the 2,500 words) that detail the data sources used. These appendices will largely consist of the material turned in as formative work, and updated in response to feedback from the course convenor. Additional details about the form, content, and structure of the report will be discussed during class sessions. The final summative report will count for the marks for the course. The report is due on Friday of Hilary Term Week 10 by 12:00pm, and should be submitted to the Examinations School. The report should also be submitted electronically by 5:00 pm on the same day (Friday of Week 10) to teaching@oii.ox.ac.uk. The report should follow the normal OII formatting guidelines. Please note that the assessment for this course is different for DPhil students who would like to take the course for credit. DPhil students should speak to the course convenor for details. Submission of Summative Assignments All coursework should be submitted in person to the Examinations School by the stated deadline. All coursework should be put in an envelope and must be addressed to The Chairman of Examiners for the MSc in Social Science of the Internet c/o The Clerk of Examination Schools, High Street. Students should also ensure they add the OII coversheet at the top of the coursework and that two copies of the coursework are submitted. Please note that all work must be single sided. An electronic copy will also need to be submitted to the department. Please note that all coursework will be marked anonymously and therefore only your candidate number is required on the coversheet. Please note that work submitted after the deadline will be processed in the standard manner and, in addition, the late submission will be reported to the Proctors' Office. If a student is concerned that they will not meet the deadline they must contact their college office or examinations school for advice. For further information on submission of assessments to the examinations school please refer to http://www.ox.ac.uk/students/academic/exams/submission/. For details on the regulations for late and non-submissions please refer to the Proctors website at https://www.admin.ox.ac.uk/proctors/examinations/candidates/. Any student failing this assessment will need to follow the rules set out in the OII Examining Conventions regarding re-submitting failed work.
Topics 1. Bibliometrics 2. Webometrics 3. Alt-metrics 4. Mapping Metrics Key to and Assignments A reading list is given below for each class. Those items marked with an asterisk () are essential reading and MUST be read by all students in preparation for the class. Items which are not marked with an asterisk are additional suggested readings which can be consulted by students with relevant interests. All assignments shown below should be submitted to the course convenor via Plato by the deadlines given above.
Week 6: Bibliometrics In this session students will be introduced to some basic principles of scientometric research. This will include an introduction to the main databases for bibliometric research (Web of Knowledge, Scopus, Google Scholar, Microsoft Academic Search), the variety of measures available (citations, co-citation, h-index and alternatives), and the limitations of these measures. Borgman, Christine L. Furner, Jonathan Scholarly communication and bibliometrics. Annual Review of Information Science and Technology 2002. 36(1): 3-72. http://dx.doi.org/10.1002/aris.1440360102 Harzing, Anne-Wil A longitudinal study of Google Scholar coverage between 2012 and 2013. 2014. Scientometrics 98(1): 565-575. Bollen J Van de Sompel H Hagberg A Chute R Hirsch, J.E. Harzing, Anne-Wil A Principal Component Analysis of 39 Scientific Impact Measures. 2009. PLoS ONE 4(6): e6022. doi:10.1371/journal.pone.0006022 An index to quantify an individual s scientific research output. 2005. PNAS 102(46): 16569-16572. www.pnas.org/cgi/doi/10.1073/pnas.0507655102 The Publish or Perish Book. Melbourne, Australia: Tarma Software Research Pty Ltd. Note Bring a laptop computer (Mac or PC) to class on which you are able to install software and access the internet. Also, you should begin thinking before the first class of a topic about which you will want to extract information during the course, so that you are ready to begin working with data right away. Formative Assignment 1 (Due Wed of Week 7) Using the topic the student has selected to pursue, produce a set of charts, tables, and graphs that support the bibliometric analysis of the topic. In collecting the data, you should use multiple databases so you can compare the results. Possible areas to explore with relation to your topic are highly cited articles, patterns of citation over time, most prominent authors, most prominent institutions, and other measures that we will discuss in week 1. The formative assignment should also include approximately 1000 words of explanation as needed.
Week 7: Webometrics (with Mike Thelwall) Webometrics is the large-scale analysis of the links, sentiment and text in blogs, web pages, social networks and digital objects on the web. This seminar will explore the use of Internet-based tools, such as commercial search engines and web crawlers, to gather web data in order to analyse the online traces of specific social phenomena (e.g., language use, news consumption) or to analyse a specific type of online behaviour (e.g., political blogging, social network communication), including sentiment analysis. The legal and ethical issues associated with methods for large-scale analysis will also be discussed. Thelwall, Mike Society on the Web, pp. 69-85 in Dutton, W. H. (ed.), The Oxford Handbook of Internet Studies. Oxford: Oxford University Press. Thelwall, M. Buckley, K. Paltoglou, G. Sentiment strength detection for the social Web. (2012). Journal of the American Society for Information Science and Technology, 63(1), 163-173. http://www.scit.wlv.ac.uk/~cm1993/papers/sentistrength2_preprint.doc Thelwall, Mike Identifying Winners and Losers: The Role of Webometrics, Chapter 1.2 in Dutton and Jeffreys, (2010) (eds) World Wide Research, pp. 45-47. Priem, Jason Hemminger, Bradley Thelwall, M., Sud, P. Wilkinson, D. Scientometrics 2.0: New metrics of scholarly impact on the social Web. First Monday, [S.l.], jul. 2010. ISSN 13960466. Available at: <http://firstmonday.org/ojs/index.php/fm/article/view/2874/2570>. Link and co-inlink network diagrams with URL citations or title mentions. (2012). Journal of the American Society for Information Science and Technology, 63(4),805-816. http://www.scit.wlv.ac.uk/~cm1993/papers/ URCitationsTitleMentionNetworks_preprint.pdf Formative Assignment 2 (Due Wed of Week 8) Using the topic the student has selected to pursue, produce a set of charts, tables, and graphs that support the webometric analysis of the topic. In collecting the data, you should use multiple data sources so you can compare the results. Possible areas to explore with relation to your topic are web links, social media data, sentiment analysis, and other measures that we will discuss in week 2. The formative assignment should also include approximately 1000 words of explanation as needed.
Week 8: Altmetrics (with Mike Thelwall) Since 2010, there has been a growing altmetrics movement, arguing for a wider variety of measures to broaden our understanding of impact. These include social media metrics, data from sources like Zotero and Mendeley, measures related to non-traditional publication means (e.g. blogging, tweeting, annotating) and modes (e.g. data publication, publication of code), and generally a flexible and agile set of approaches to measuring impacts. In this session, we will discuss a variety of these methods, and students will start to use altmetric methods to gather data related to their projects. Almetrics Blog Altmetrics manifesto. Online at http://altmetrics.org/manifesto/ Piwowar, Heather Value All Research Products. 2013. Nature 493 (10 Jan 2013): 159. Thelwall, Mike Haustein, Stephanie Lariviere, Vincent Sugimoto, Cassidy Eccles, Kathryn Thelwall, Mike Meyer, Eric T. Samoilenko, Anna Yasseri, Taha Shuai X. Pepe A. Bollen, J. Mohammadi, Ehsan Thelwall, Mike Do Altmetrics Work? Twitter and Ten Other Social Web Services. 2013. PLoS ONE 8(5): e64841. doi:10.1371/journal.pone.0064841. Measuring the Web Impact of Digitised Scholarly Resources. 2012. Journal of Documentation 68(4):512-526. "The distorted mirror of Wikipedia: a quantitative analysis of Wikipedia coverage of academics." 2013. arxiv preprint arxiv:1310.8508 How the Scientific Community Reacts to Newly Submitted Preprints: Article Downloads, Twitter Mentions, and Citations. 2012. PLoS ONE 7(11): e47523. doi:10.1371/journal.pone.0047523 "Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows." 2014. Journal of the American Society for Information Science and Technology Formative Assignment 3 (Due Wed of Week 9) Using the topic the student has selected to pursue, produce a set of charts, tables, and graphs that support the altmetric analysis of the topic. In collecting the data, you should use multiple data sources so you can compare the results. Possible areas to explore with relation to your topic are alternative measures of prominence from the social web, preprint servers, readership data, Wikipedia prominence, and other measures that we will discuss in week 3. The formative assignment should also include approximately 1000 words of explanation as needed.
Week 9: Mapping Science The visual display of metrics can be a useful means of finding patterns in data, but also for communicating results. In this session, we will discuss various approaches to visualizing and displaying data, and work with tools to enable students to create visualizations of their own data. Bollen, J. Van de Sompel, H. Clickstream Data Yields High-Resolution Maps of Science. PLoS ONE. 2009. 4(3): e4803. Borner, K. Scharnhorst, A. Shiffrin, R. Borner, K. Van den Besselaar, P. Heimeriks, G. Leydesdorff, L. de Moya-Anegón, F. Guerrero-Bote, V. "Visual conceptualizations and models of science." Journal of Informetrics 2009. 3(3): 161-172. "Mapping knowledge domains." Proceedings of the National Academy of Sciences of the United States of America. 2004. 101.Suppl 1: 5183-5185. Mapping research topics using word-reference co-occurrences: A method and an exploratory case study. 2006. Scientometrics, 68(3): 377-393. Journal Maps, Interactive Overlays, and the Measurement of Interdisciplinarity on the Basis of Scopus Data (1996-2012). arxiv:1310.4966 Optional work Using the topic the student has selected to pursue, produce novel visualizations (that go beyond simple charts, tables, and graphs) of the data you have been working with this term.