EDM Workshop: Writing Analytics, Data Mining, & Writing Studies

Overview

EDM Workshop: Writing Analytics, Data Mining, & Writing Studies
June 29, 2016, Raleigh, NC, USA
9th International Conference on Educational Data Mining (EDM 2016)

Writing Analytics, Data Mining, & Writing Studies, a preconference workshop at EDM 2016, aims to generate cross-disciplinary research among writing program directors and faculty, computational linguists, computer scientists, and educational measurement specialists. The primary goal of this workshop is to facilitate a research community around the topic of large-scale data analysis with a particular focus on writing studies, data mining, and analytics.

The workshop

  • aims to generate cross-disciplinary research among writing program directors and faculty, computational linguists, computer scientists, and educational measurement specialists
  • questions ways writing analytics and data mining can be used to improve on existing methods for responding to and assessing student writing
  • invites researchers in the domains of data mining, writing analytics, and writing studies to engage in a creative interdisciplinary exploration of how digitally based writing analytics might improve students’ cognitive, intrapersonal and interpersonal competencies as writers, and also provide new analytic tools for assessing this improvement
  • invites participants to brainstorm about ways analytics can improve document critique, peer review, and writing program assessment.

Workshop Resources

Call for Participation

CALL FOR PARTICIPATION

EDM Workshop: Writing Analytics, Data Mining, & Writing Studies
June 29, 2016, Raleigh, NC, USA
9th International Conference on Educational Data Mining (EDM 2016)

We invite submissions to a pre-conference workshop on Writing Analytics, Data Mining, & Writing Studies at EDM 2016, which will be held at Raleigh, NC on June 29, 2016. This workshop is a precursor to EDM 2016 (The 9th International Conference on Education Data Mining), a leading international forum for high-quality research that leverages data and data science to answer research questions that shed light on the learning process.writing-data-and-writing-image_final

The primary goal of this workshop is to facilitate a research community around the topic of large-scale data analysis with a particular focus on writing studies, data mining, and analytics. The workshop hopes to generate cross-disciplinary research among writing program directors and faculty, computational linguists, computer scientists, and educational measurement specialists. This workshop questions ways writing analytics and data mining can be used to improve on existing methods for responding to and assessing student writing. This workshop invites researchers in the domains of data mining, writing analytics, and writing studies to engage in a creative interdisciplinary exploration of how digitally based writing analytics might improve students’ cognitive, intrapersonal and interpersonal competencies as writers, and also provide new analytic tools for assessing this improvement. Workshop participants will be introduced to current trends in data mining and writing analytics. In summary, the workshop will provide an overview of development work on writing analytics and invite participants to brainstorm about ways analytics can improve document critique, peer review, and writing program assessment.

Please share your related research with us. Sample topics may include

  • How can data mining and analytics be leveraged to better meet the needs of students and educational institutions?
  • What are the best practices for adapting the state-of-the-art data mining approaches to the educational domain, with specific attention to teaching and assessing writing?
  • How can researchers detect and assess students’ intrapersonal domain characteristics while engaging the writing construct?
  • How can researchers detect and assess students’ interpersonal domain characteristics while engaging the writing construct?
  • What broad cognitive domain characteristics best capture the writing construct, and how are these characteristics modified by task?
  • How may the writing construct best be modeled according to cognitive, intrapersonal, and interpersonal domains?
  • For assessing writing, automated grading, automated commenting, natural language or textual data processing:
    • What are applications of massive parallel computations?
    • What are current advances and future directions in the artificial intelligence field?
    • What methods, tools or big data platforms are more efficient?
    • What are effective pre-processing techniques, e.g. for the Extract/Transform/Load phase?
    • What are successful evaluation and validation methods?
  • How can data mining and writing analytics inform peer review practices?
  • What genres of real-time reporting can meet evidence demands of reliability, validity, and fairness while providing actionable information to students leading to improved writing performance?
  • How can researchers visualize writing analytics to make feedback more meaningful for students?

Due Dates

May 1: Paper submissions
May 8: Notification to authors
May 31: Final papers due

Publication & Submission Guidelines
Workshop proceedings will be published on the CEUR Workshop Proceedings site and include all accepted workshop submissions. Submissions should be prepared with EDM’s templates: Word, LaTex (zip file) and submitted via EDM 2016’s EasyChair.

Full Papers: 6 to 8 pages. Original, substantive, mature and unpublished work.
Short Papers: 3 to 5 pages. This includes early stage, less developed works in progress.
Posters, Demos: 2 pages.

Workshop Organizers

  • Val Ross, University of Pennsylvania, Director of Critical Writing Program, https://www.english.upenn.edu/people/valerie-ross
  • Alex Rudniy, Fairleigh Dickinson University, Assistant Professor of Computer Science, https://www.linkedin.com/in/alex-rudniy-4698ba2b
  • Dave Eubanks, Assistant Vice President, Furman University, https://www.linkedin.com/in/david-eubanks
  • Joe Moxley, University of South Florida, Director of First-Year Composition, http://joemoxley.org

Program Committee

  • Chris Anson, North Carolina State University, Director of Campus Writing and Speaking Program, http://ncsu.academia.edu/ChrisAnson
  • Laura Aull, Wake Forest University, Assistant Professor of English and Linguistics, http://wfu.academia.edu/LauraAull
  • Denise Comer, Duke, Director of Writing, https://www.linkedin.com/in/denise-comer-2407666?
  • Norbert Elliot, NJIT, Professor Emeritus, https://www.linkedin.com/in/norbert-elliot-1712a89
  • Ann Gere, University of Michigan, Director of Writing, Sweetland Writing Center, http://www.soe.umich.edu/people/profile/anne_ruggles_gere/
  • Larry Hall, USF, Distinguished University Professor, Computer Science, http://morden.cse.usf.edu/ailab/hall.html
  • Asko Kauppinen, Malmö University, The Writing Unit, Director of Research, http://forskning.mah.se/en/id/imaska
  • Andrew Krumm, SRI International < https://www.linkedin.com/in/andrew-krumm-5aa631a?
  • Roberto Martinez-Maldonado, University of Technology, Postdoctoral Research Associate, http://www.uts.edu.au/staff/roberto.martinez-maldonado
  • Alla Rozovskaya, Virginia Tech, Assistant Professor of Computer Science, https://www.linkedin.com/in/alla-rozovskaya-08908a54
  • Djuddah A.J. Leijen, University of Tartu, Head, Centre for Academic Writing and Communication, https://www.researchgate.net/profile/Djuddah_Leijen
  • Ravi Rao, Fairleigh Dickinson University, https://www.linkedin.com/in/drravirao
  • Anna Wärnsby, Malmö University, The Writing Unit, Director of Research, http://forskning.mah.se/en/id/imanwa

Background
Current measures of coaching and assessing student writing, while time consuming and well intentioned, fail to provide students with the feedback they need to improve as writers and peer reviewers. After all, current assessments of students’ writing competencies have identified problems with students’ reading, research, collaboration, and communication competencies: in the U.S. the College Board determined that 57% of SAT takers do not qualify as college ready; the ACT found 31% of high school graduates “did not meet any of the ACT College Readiness Benchmarks”; the NAEP Writing Report found 73% of 12th graders received scores of Below Basic or Basic as opposed to Proficient or Advanced in 2011; the Programme for International Student Assessment concluded that the U.S. literacy rate fell from 10th to 20th in the latest study on global rankings.

Digital tools such as My Reviewers that enable instructors to grade and comment on student papers and peer reviews online are transforming how instructors and students critique documents and have the potential to transform how writing and writing programs are assessed. Beyond profoundly altering how faculty and students respond to writing, these tools aggregate e-portfolios, facilitate distributive evaluation, and archive data that allow researchers to mine texts and map student outcomes in order to produce analytics that inform users, researchers, and administrators. Rather than limit assessment to cognitive measures, these toolsets facilitate gathering authentic assessment information about students’ intrapersonal and interpersonal competencies.

PDF

Organizers

Val Ross

University of Pennsylvania, Director of Critical Writing Program,

https://www.english.upenn.edu/people/valerie-ross

Valerie Ross received her doctorate from the University of Wisconsin-Milwaukee in English and Modern Studies, and has published and presented in the fields of writing studies, biography, critical theory, cultural studies, and nineteenth-century American literature. Her current research interests include cinema and rhetoric. Before joining Penn, Dr. Ross was an Assistant Professor of English at Miami University.  She directs the Critical Writing Program for the Center for Programs in Contemporary Writing.

Alex Rudniy

Fairleigh Dickinson University, Assistant Professor of Computer Science

https://www.linkedin.com/in/alex-rudniy-4698ba2b

Alex Rudniy received his B.S. and M.S. at the National University of Radioelectronics in Ukraine. He went on to earn his Ph.D. from the New Jersey Institute of Technology. Before becoming an Assistant Professor in Computer Science at Fairleigh Dickinson University, he worked as a Data Manager, Research Assessment Analyst, and Research Associate.

Dave Eubanks

Assistant Vice President, Furman University

https://www.linkedin.com/in/david-eubanks

David Eubanks is the Assistant Vice President for Assessment and Institutional Effectiveness at Furman University, a private liberal arts university.

Joe Moxley

University of South Florida, Director of First-Year Composition

http://joemoxley.org

Joe Moxley is the Founder of Writing Commons, a free alternative to expensive writing textbooks. Peer-reviewed, Writing Commons provides open access to over a thousand webtexts, making it a viable choice as the required textbook for composition, professional and technical writing, creative nonfiction, and creative-writing courses. Moxley is also the Director of First-Year Composition at the University of South Florida, a Research 1 university.

Related Research:

  1. Anson, C., Moxley, J. Lejen, D., Finnegan, D., Warnsby, A., and Kauppinen, A. 2015. Theorizing community rubrics: limits, research, and case studies. In EATAW 2015: 8th Biennial Conference of the European Association for the Teaching of Academic Writing. (Tallinn University, Estonia, June 15-17, 2015).
  2. ACT. 2013. The Condition of College & Career Readiness 2013. ACT National Report: ACT Inc., Aurora, CO. chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/https://www.act.org/research/policymakers/cccr13/pdf/CCCR13-NationalReadinessRpt.pdf.
  3. Anson, C., Moxley, J. Lejen, D., Finnegan, D., Warnsby, A., and Kauppinen, A. 2015. Theorizing community rubrics: limits, research, and case studies. In EATAW 2015: 8th Biennial Conference of the European Association for the Teaching of Academic Writing. (Tallinn University, Estonia, June 15-17, 2015).
  4. College Board. 2013. 2013 SAT Report on College and Career Readiness. College Board Report. The College Board, New York, NY.
  5. Dixon, Z., & Moxley, J. M. 2013. Everything is illuminated: what big data can tell us about teacher commentary. Assessing Writing 18 (2013), 241-256. DOI= http://dx.doi.org/10.1016/j.asw.2013.08.002.
  6. Langbehn, K., McIntyre, M. & Moxley, J. M. 2013. Re-mediating writing program assessment. In Digital Writing Assessment & Evaluation. Heidi A. McKee and Danielle Nicole DeVoss, Eds. Computers and Composition Digital Press, Logan, UT. http://ccdigitalpress.org/dwae/13_langbehn.html.
  7. Moxley, J. M. 2013. Big data, learning analytics, and social assessment. Journal of Writing Assessment 6, 1 (2013),  n. pag. http://www.journalofwritingassessment.org/article.php?article=68.
  8. Moxley, J. 2015. My Reviewers: The development and user-centered design of an internally-produced document review, assessment, and e-portfolio tool. In The ACM Special Interest Group on the Design of Communication (ACM SIGDOC) (Limerick, Ireland, July 16-17, 2015).  http://sigdoc.acm.org/conference/2015/.
  9. Moxley, J. 2015. Peer review and document review in STEM courses. In ProCOMM 2015. IEEE International Professional and Communication Conference. (Limerick, Ireland, July 12-15, 2015). http://pcs.ieee.org/procomm2015/.
  10. Moxley, J. 2015. Statistical, Predictive, and Discourse Analysis of 78,000 Peer Reviews.  In USF Colloquium: Digital Writing Tools for Global Citizens. (Tampa, FL., January 16, 2015). http://toolsforwriters.com/wp-content/uploads/2014/09/2015ColloquiumDigitalWritingToolsProgram.pdf.
  11. Moxley, J. M. 2014. Using digital tools to facilitate and assess the development 21st century literacy competencies in scientific and technical writing programs. In Conference on Programs in Technical and Scientific Communication. (University of Colorado at Colorado Springs, September 25-27, 2014).
  12. Moxley, J. 2015. Using digital tools to improve feedback on student writing, peer review, and writing program assessment. In 21st Century Academic Forum at Harvard. Teaching, Learning, and Research in the ‘Just Google It’ Age. (Harvard, MA, March 9, 2015).
  13. Moxley, J. & Eubanks, D. In Press. On keeping score: instructors’ vs. students’ rubric ratings of 46,689 essays. WPA: Writing Program Administration. (In Press).
  14. Moxley, J., Ross, V., Lane, S., Donahue, C., Anson, C., and Rudniy, A. 2015. NSF PRIME: the role of instructor and peer feedback in improving the cognitive, interpersonal, and intrapersonal competencies of student writers in STEM courses. NSF Grant. Award number: 1544239.
  15. National Research Council. 2013. Frontiers in Massive Data Analysis. Washington, D.C.: The National Academies Press.
  16. Programme for International Student Assessment. 2012. Programme for International Student Assessment Results from PISA 2012: United States. OECD. chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/http://www.oecd.org/pisa/keyfindings/PISA-2012-results-US.pdf
  17. Tackitt, A., Moxley, J., and Eubanks, D. 2015. Signifying scores: instructor rating as an assessment measure. Assessing Writing. (Submitted).

Program

9:00 – 9:30 Introduction: Writing Analytics, Massive Data Analysis, and My Reviewers

9:30 10:00 Paper 1: Mapping Writing Analytics
Joe Moxley and Katie Walkup, University of South Florida
This paper endeavors to define and map Writing Analytics (WA) as an academic field. The affordances of digital tools have enabled us to reimagine WA’s meanings and applications. The authors use the metaphor of mapping to understand the tensions and successes navigated by researchers and practitioners and to chart new ways in which this field can benefit the domains of academia, business, and culture. Approaching WA from an interdisciplinary perspective allows the field to consider new research questions.

This workship engages participants in a collaborative effort to define WAs. After examining the following image, participants are encouraged to create their own heuristics in Google Drawings, Mindomo, or Coggle. Please share these heuristics on our Google Doc: https://docs.google.com/document/d/10ZLbdsIE20HBpZjh687nQyRpHtjjw4RHHjueZcH0NZk/edit?usp=sharingnewest coolest image

10:00 – 10:30 Paper 2: Collaborative Review in Writing Analytics: N-Gram Analysis of Instructor and Student Comments
Alex Rudniy1 and Norbert Elliot2
1 Fairleigh Dickinson University, 2 New Jersey Institute of Technology
The purpose of this paper is to explore the use of n-gram analysis to analyze instructor and student comments elicited within My Reviewers, a web-based learning environment. Shown to be informative in a wide variety of applications, n-gram analysis is of interest in determining concept proliferation in topics, purposes, terminologies, and rubrics used in writing courses. As the present study demonstrates, unigram, bigram, digram, trigram, fourgram, and fivegram analytic methods reveal important information about instructor and student use of concepts; in turn, such analysis holds the potential to lead to precise and actionable revision behaviors.

10:30 – 11:00 Coffee Break

11:00 – 11:30 Paper 3: Corpus Methods and Textual Visualization to Enhance Learning in Core Writing Courses
David Kaufer and Suguru Ishizaki, Carnegie Mellon University
Writing tasks require countless composing decisions that are typically beyond the conscious grasp of writers. Much of the skill of being “text-aware” is to understand that texts produced from classroom assignments are not just composed of words and sentences but of highly structured and often highly predictive composing decisions. However, the decision-making underlying writing is an extremely abstract idea that is hard to make tangible for students. Although a significant number of pedagogical approaches has been investigated in the past three decades, the means to help students acquire more tangible understanding and control of their composing decisions has not been addressed. We propose to address this gap by developing a corpus-based learning tool to help students notice and reflect on composition decisions in their writing and to become resultantly more self-aware and reflective writers. This approach builds on an existing corpus-based text analysis tool called DocuScope, which for over a decade was successfully used for these purposes in our graduate writing course. The goal of this project is to extend this approach to support the core writing courses at our university.

11:30 12:00 Paper 4, Weighted Log-odds-ratio, Informative Dirichlet Prior Method to Enhance Peer Review Feedback for Low- and High-scoring College Students in a Required First-year Writing Program
Valerie Ross, Mark Liberman, Lan Ngo, Rodger LeGrand, University of Pennsylvania
The purpose of this paper is to use a weighted log-­odds-­ratio, informative Dirichlet prior method (“bag of words” approach) to analyze student comments and scores posed to My Reviewers, a web­-based tool designed to collect student writing as well as their peers’ comments and scores of their colleagues’ drafts. As this preliminary study suggests, the use of this method shows that lower performing writers might be receiving kinds of feedback generally viewed as counterproductive in the field of writing studies. Other implications include the potential for evaluating the effectiveness of such feedback on revision, attitudes toward writing, and motivation; and for training students and teachers in effective feedback approaches.

12:00 – 12:30 Collaborative Activity, Challenges and Benefits in Corpus Methods
In this collaborative activity, we will identify the challenges and benefits to corpus techniques.

12:30 – 2:00 Lunch

2:00 – 2:30 Paper 5, First-Year Composition as “Big Data”: Examining Student Revisions at Scale
Chris Holcomb and Duncan Buell, University of South Carolina
Approaching First-Year Composition (FYC) as a “big data” phenomenon, we have prototyped software to study revision in a large corpus of student papers and thus to address a question central to Composition and Rhetoric scholarship: “What role does revision play in students’ writing processes?” After running our program on a corpus of student writing, we see that our computational analysis challenges past research on revision and extends the methodological reach of Composition and Rhetoric to include “big data” analytics.

2:30 to 3:00 Paper 6, Writing Transfer as a Framework for Big Data and Writing Analytics Research
Denise Comer, Duke University
In this poster presentation, the author will use the frame of writing transfer to explore how researchers can transfer strategies, approaches, and knowledge about writing gained from big-data writing analytics to other writing pedagogy contexts. Comer will share methods and results from the following four big-data research projects stemming from research in her writing based Massive Open Online Course: 1. Big data and writing assessment; 2. Big-data, writing, and peer-to-peer interactions; 3. Big-data, writing, and negativity; and 4. Big data, peer-review and transfer. These project overviews will be presented as a means of exploring the affordances and limitations of using big-data writing analytics to improve the teaching and learning of writing.

3:30 – 4:00 Coffee Break

4:00 – 5:00 Collaborative Activity, Mapping Writing Analytics
In this final collaborative activity, we will work collaboratively to define and visually map the new field of Writing Analytics, discuss possible collaborations, and imagine the impossible–new ways to structure student success for diverse student audiences.