Scan&Score

Instructions for Using the Application

[Nov 23, 2009 14:09] Web access to Microsoft Live@edu accounts now works.

Using the Scan&Score Application

Scan&Score is a Web-based exam scoring system that will:

  • accept data files from external sources,
  • provide methods for scoring and analyzing the input data, and
  • provide the ability for the user to retrieve the final analysis.

The Scan&Score score application will support the exam scanning and scoring services for all University professors and course directors. The application allows data produced by the Opscan 6 Scantron scanner and other third-party applications to be analyzed, scored, sorted and saved in a secure environment.

Scan&Score Feature Set

Loading the initial exam data

Exam data is loaded into the Scan&Score application from a formatted data file typically produced by scanning a Scantron (bubble sheet). Scan&Score will also accept data files from other sources as long as they are properly formatted.

The ability to load exam data into the application is limited to administrators. This feature will not be described in this document. Administrators are responsible for loading the exam data files under the appropriate exam owner.

Accessing the exam data

The exam owner has the ability to retrieve the raw exam data, modify the answer key, score the exam, subset the exam by students or questions. The user can be granted the rights to delete their exams by the site administrator. The exam owner may also grant access to their exams to other users.

Figure 1 shows the application header and three key features that appear on every page of the application.

figure 1

Figure 1 Standard application bar

  1. Item 1 is known as "bread crumbs" and it shows you where you are in the application. You can move to previous pages from page to page by clicking on the bread crumb links.
  2. The Online Help button will display a pop-up window with help specific to the active page.
  3. Logout will clear your session variables. It is recommended that after logging out, you close your browser when working from a public PC.

When a professor or a TA logs into the Scan&Score application, they will see a screen like the one shown in Figure 2 with their active exams listed. To view archived exams click on the "View Archived Exams" checkbox.

figure 2

Figure 2 Initial Professor/TA screen – Select Exam Page

  1. Professor's name. Typically only one name in dropdown, but if the user has access to multiple professors' exams there will be more than one name to select from.
  2. Manage Exam Access is used to grant or revoke access to a TA or professor.
  3. Select is used to choose the exam in the associated row to modify grades, create subset, modify student data, view grades and statistical data, view raw data and export to UVaCollab.
  4. Delete will delete the associated exam.

The Exam Home Page

The Exam Home Page is designed to give the user access to all features for a selected exam. All features and functions of the Exam Home Page are associated with the exam selected by the user.

figure 3

Figure 3 – Exam Home Page

  1. This item indicates which exam was selected. The user can see the exam title, course number, course term and date exam was given. The user should verify that they are looking at the correct exam.
  2. The Exam Management functions allow the user to modify the answer key, correct misbubbled computing IDs, create a copy of the exam, change exam details, split the exam into two exams, change a student answer for a particular question and create subsets by either students or questions.
  3. The Reports functions provide student grades and other statistical information. A detailed description of each report type is provided later in this document.
  4. The Transfers/Downloads functions allow a user to obtain a copy of the raw data file and/or create a file to upload to UVaCollab.

Exam Management

Answer Key

  1. Click the Modify Mulitple Weights link to change the weighted value of one or more question.
  2. Click the Edit link to change the acceptable answers for a question. Can choose to accept more than one answer, change the weight for the selected question or click the deleted check box to exclude question from exam.

figure 4

Figure 4 – Modify answer key

Computing IDs

Click the Edit link to change a student's computing ID. This would be used when a student failed to correctly bubble in their computing ID on the original Scantron sheet.

Click on the Show Problem Id's (1) to only show those IDs which appear to have an error. This would include IDs with embedded blanks, blank IDs and IDs which were improperly bubbled in.

figure 5

Figure 5 – Student Computing IDs

Copy Exam

This page allows a user to make a copy of the original exam by selecting one or more questions to include. The answer key and student answers are copied to a new exam. The original exam is not impacted by the copy exam function.

  1. Specify a new exam name
  2. Click the Save button to save changes

Figure 6 shows a copy of questions 1 through 10 being created as "Top10Questions."

figure 6

Figure 6 – Copy Exam Page

Edit Exam

Allows the user to change the exam identifying data. The initial data is entered by the scanner from the sheet provided by the exam owner. The exam name and exam date are required fields.

figure 7

Figure 7 – Edit Exam Page

Split Exam

  1. Specify the name of the two resulting exams.
  2. Select the last question number to be included in the first exam.

Figure 8 shows an exam that is being split after question 20. In this example, the exam "First20Questions" will include 1–20 and the exam "RemainingQuestions" will include 21–last question number.

figure 8

Figure 8 – Split Exam Page

Student Answers

  1. Choose student of interest from dropdown list.
  2. The Scan Number is printed on the scantron sheet. It is used to match to the actual sheet.
  3. The Edit link allows you to change the student's answer. If the student filled in multiple answers, the answer was unreadable or the answer appears blank, the appropriate check box will be marked. The user can pick the new answer from the Answer dropdown box.

figure 9

Figure 9 – Student Answers

Subset By Students

Subsetting by students can be used to break out different groups of students. If the class includes students who are taking the course as part of their major as well as students who are taking the course as an elective, the professor may want to compare the two groups to the grades overall. This could be used to separate by major or year in school.

  1. Specify a subset name used to identify the new result set
  2. Select a group of students to be included in the subset

figure 10

Figure 10 – Subset by Student

Subset By Questions

Subsetting by question can be used to see how the students did on questions relevant to a particular subject. For example if questions 1–9 are testing students on material covered in class, and 10–20 are testing students on material from the text, the professor could subset the exam into two sets and see if the scores vary between sets.

  1. Specify the name in which to identify the created subset.
  2. Specify the questions to be included in the subset.

figure 11

Figure 11 – Subset by Question

Reports

The different reports provide the student grades, exam statistics and error reports. Each report appears in a separate window and gives the user the ability to page through the report, sort on different columns and export the results to Excel or PDF. Figure 12 is the standard report header.

  1. Click on the column header to sort data.
  2. Select the Export to CSV link to export the data to a CSV file.
  3. Select the Export to PDF link to export the data to a PDF file.

figure 12

Figure 12 – Standard Report Header

Exam Scores Report

  1. Total Correct represents the number of correct answers.
  2. Total Points represents the number of points received. This will equal the Total Correct if all questions have a weight of 1.
  3. The number of exams indicates the number of students who have a grade. This value is used in calculating the statistics.
  4. Various statistical data found on the last page of the Scores Report.

figure 13

Figure 13 – Exam Scores Report

Frequency Report

From left to right shows Number of Correct answers, percent correct, how many students answered that number correctly, what percent of the students from the class answered that many correctly and a running cumulative percent.

  1. Click on View Graph for a graphical representation of frequency data.

figure 14

Figure 14 – Frequency Report

Item Analysis

An item analysis aids in determining which exam questions were weak or represented unsatisfactory levels of understanding. The analysis includes for every question the question number, the correct choice, the number of omissions of that item, the numbers of persons marking each choice (shown on first row), the percentage of the class selecting each choice (second row) and a Pearson product moment correlation between each choice and total scores.

Results for the correct choice (marked by an asterisk) are most important. Good exam questions have most or all of the following characteristics:

  • Between 30 and 85 percent of examinees should answer correctly. If the item is too hard or too easy it contributes relatively little toward ranking examinees according to their knowledge.
  • Each of the answer choices should attract at least some of the examinees. If a wrong choice is so obvious that no one selects it, student testing time is saved by omitting it on future exams. There is no need to have the same number of choices for all items.
  • The correlations between choices and total score should be positive (preferably .3 or higher) for the right choice and negative for the wrong choices. This outcome would indicate that better students tended to get the item right. If a positive correlation occurs for the wrong answer, it indicates that better students were misled into selecting the wrong choice. A poor item correlation can also alert the instructor to a possible mistake in filling out the key.
  1. Click on the plus (+) to show the top and bottom 27% when not visible or to hide it when visible.

figure 15

Figure 15 – Item Analysis Report

Load Log

Shows errors that occurred when the exam was initially loaded. Errors that can occur are duplicate IDs or blank IDs. In both cases the records are loaded, but the scansheet number is appended to the ID to make it unique.

Missed Questions

Lists for each student a row per missed question. The report shows the student ID, test number if specified, the number of the missed question, the correct answer for that question and the students answer. Report is ordered by student ID and then question number.

figure 16

Figure 16 – Missed Questions Report

Score Rank

Shows how a student did in relation to the rest of the class. It displays the ranking of the student's exam scores from highest to lowest. Total Correct is the correct number of questions answered. Total Points reflects the weight of the question.

figure 17

Figure 17 – Score Rank Report

Student Answers

Show the answer key, weight of each answer and the answer selected by each student.

  1. The Answers section shows the answer(s) for each question followed by the weight given to each question. If the question is being excluded from the grade it is denoted with double asterisks (**). This section reflects changes made to the answer key through the Scan&Score application.
  2. The Answer Key section shows the answer from the original source.
  3. The student answers section shows the answer for each question for each student. Changes made using the Scan&Score application are reflected in the output.

figure 18

Figure 18 – Student Answers Report

  1. Question 1 has been deleted and will not be included in the grading.
  2. Question 5 has a weight of 2 points.
  3. Question 10 two correct answers (BC).

figure 19

Figure 19 – Detail of answer key

Understanding the Output

Statistics, including number of students, number of questions means, standard deviations, high and low exam scores and KR20 reliability estimate.

Median Number Right–the midpoint of the distribution where half the scores are above and half below.
Number of Students–how many students took the exam.
Reliability Estimate (KR-20)–an estimate of the correlation between the scores on the exam and the scores which would be obtained if students were given a similar exam on the same material. Satisfactory levels range from .5 for exams of less than 15 items to .8 and up for longer exams. The method for computing this estimate assumes that essentially a single area of knowledge is being tested. See Testing Memo 2 for more information on reliability scores.
Standard Deviation of Number Right–a measure of how spread out the scores are on either side of the mean number right.
Standard Error of Measurement–an estimate of how far a student's observed score may be from a true score, or the mean score that would be earned on many independent administrations of the exam.
T-Score–is the number of standard deviations from the mean and may be used for "curving" grades.

Transfers/Downloads

Users should heed the warning below when downloading student grades.

figure 20

Figure 20 – File Download Warning

  1. Raw Data File allows the user to download the data that was initially loaded. This is the file that originated from the Scantron sheets or from an external program.
  2. UVaCollab: Upload Grades allows the user to generate a file that is automatically uploaded to your My workspaces > Resources in UVaCollab.

Frequently Asked Questions

Q: I'm going to give an exam. Where do I get blank Scantron forms? Is there any charge? Who can use your services?
A: There is no charge for the sheets or for grading an exam. Anyone associated with the University of Virginia may use our services. Scanning Process >>

Q: Do the students really need to use No. 2 pencils?
A: Yes, No. 2 pencils work best; most inks are transparent and do not contain enough graphite to block the light and so are not "read" by our scanner. Proper Bubble Sheet Handling/Marking >>

Q: My exam ends after 5PM. Can I still drop off my exam?
A: Yes, we have a drop off slot in our door for after hours exams. Exam cover sheets and campus mailing envelopes are available too. Scanning Process >>

Q: How long does it take to get back my exam results?
A: This will depend on when the exam was dropped off and what else we are processing. During the regular semesters you will usually get the results anywhere from a half hour to a few hours. During final exam periods response time can depend on how many exams were brought in ahead of yours but we still try for a 2 hour maximum return time. Scanning Process >>

Q: How do I post my grades for students to view?
A: Through UVaCollab.

Q: I miscoded my answer key. Do I need to bring you a new key?
A: No, you can edit your answer key online. See Exam Management / Edit Exam.

Q: Why can't I have more than one correct answer for a question?
A: You can edit the answer key online and allow multiple answers for a question. You cannot mark multiple correct answers on a Scantron sheet.

Q: I accidentally used two correct answers for a question. Can I mark both on my key as correct?
A: No. The easiest way to handle this is to modify the answer key online.

Q: (1) I have 20 questions on my exam and a bonus question. OR (2) Some of my questions are worth more than others. What kind of output will I get?
A: Once the exam has been loaded into the system you can edit the weight for the question. The final exam scores will reflect the weighted values.

Q: I don't know what some of the numbers on my printout mean.
A: Explanations can be found for the various statistical values, such as Reliability Estimate (KR-20), T-Score, etc. See more in Reports.

Q: I want to put my exam scores into an Excel spreadsheet.
A: Use the export option when viewing the report. Remember that the exam data needs to be protected.

© 2009 by the Rector and Visitors of the University of Virginia.

The information contained on the University of Virginia’s Department of Information Technology and Communication (ITC) website is provided as a public service with the understanding that ITC makes no representations or warranties, either expressed or implied, concerning the accuracy, completeness, reliability or suitability of the information, including warrantees of title, non-infringement of copyright or patent rights of others. These pages are expected to represent the University of Virginia community and the State of Virginia in a professional manner in accordance with the University of Virginia’s Computing Policies.