Qualifications

If you need a document translated from Czech into English, you don’t want just anybody to perform the translation. You’d prefer a translator, who was born and raised in Prague, or a translator with a degree in Slavic languages from a reputable university, or a translator with a reputation for providing quality translations to other Requesters. A Slavic languages student might be interested in translating your text for you, but you might prefer an experienced professional, especially if you are willing to pay professional rates. Mechanical Turk lets you describe the kind of Turk who would best perform your HIT. Only Turks that meet your requirements will be allowed to accept and complete your HITs.

Similarly, as a Turk, you may want to limit the HITs you see to those from Requesters with certain qualities, or with a good reputation among other Turks performing similar kinds of tasks. Mechanical Turk lets you describe the kind of Requester you’d prefer to work with as part of your search criteria. Only HITs from Requesters that meet your requirements will show up in the search results.

Turks and Requesters have qualification scores that describe their skills, credentials, and reputation. A score is a numeric or yes/no value: .This Turk’s SAT math score is 770. This Turk has a degree in Slavic languages from the University of Washington. This Requester has submitted and approved 120 HITs to date.

HITs have qualification requirements that the Turk must meet to accept the Requester’s

HIT. Similarly, searches for HITs may include requirements the Requester must meet to have their HITs show up in the results. A requirement is a conditional statement about a score: A Turk’s SAT math score must be 750 or higher. A Turk must have a degree in Slavic languages from the University of Washington. A Requester must have submitted and approved at least 100 HITs.

When someone assigns a score to someone else, they are making a statement about that person. The author of such a statement is just as important as what the statement says:

.A Turk’s SAT math score is 770 according to the Princeton Review. Requirements must be equally explicit: .A Turk’s SAT math score according to the Princeton Review must be 750 or higher. A similar statement made by someone who is not considered an authority on the subject would not be as relevant: a Turk’s SAT math score according to Joe’s Standardized Testing Emporium might not make for a useful requirement.

Mechanical Turk assigns scores to users based on their statistics. These scores represent a user’s experience with the system, and are adjusted over time. Scores assigned by Mechanical Turk provide an objective view of how users participate in the system.

For example, Turks have scores for the number of HITs completed, how many of those were approved by their Requesters, and how many HITs were abandoned. Requesters have scores for the number of HITs submitted, how many assignments for those HITs were approved, and how many assignments are still awaiting approval.

In addition to scores based on statistics, Mechanical Turk provides a way for users to rate other users on their performance. Turks request to be rated on how well they perform certain skills, and build reputations for performing tasks that use those skills.

When a Turk asks to be rated for a particular skill, such as .can translate Czech to English, Mechanical Turk creates a qualification score and begins tracking ratings. As the Turk completes HITs that require that skill, Requesters are asked to rate their performance.

Mechanical Turk sets the score equal to the average of all ratings collected. Turks can also rate Requesters on a set of skills common to Requesters, such as their ability to communicate with Turks, or the accuracy of their HIT descriptions.

Anyone can be the author of a qualification score. Universities, standardized testing institutions, and certification agencies may award Turks and Requesters with scores based on degrees or grades they have earned. Third-party verification agencies may award scores based on proof of credentials in exchange for a fee; Requesters could choose to accept scores assigned by such agencies known to be reputable.

Requesters can use qualification scores and requirements to screen potential Turks for their HITs, as follows: The Requester invites potential Turks to take a test to demonstrate a skill, or submit samples of their work. The Requester reviews their performance and assigns scores accordingly. The Requester can then create HITs that require Turks have passed the screening process.

With millions of possible ways to describe a person’s abilities, and millions of potential authors describing those abilities, the universe of possible qualifications is vast. It is in Turks’ and Requesters’ best interests to establish a standard vocabulary of qualifications and authorities trusted to award scores: Turks want qualification scores that most Requesters are using for their HITs and Requesters want to specify requirements that most Turks meet. Mechanical Turk provides a powerful search engine for determining which qualifications are most commonly used for certain kinds of HITs. Other features of the turk.com console, such as displays of .featured. Qualifications determined by Mturk.com algorithms, promote the use of common qualifications.

Only the author of a qualification score can change or remove it. If you are awarded a score you don’t like, you may contact the author that assigned the score to have it changed or removed. Note that a score assigned by someone not considered an authority on the subject (such as an irate user) is not likely to have any effect: the author’s name must be specified with the qualification to create a requirement.

Mechanical Turk is the author of automatic scores and ratings scores. Since ratings are added to your profile by request, they can also be removed by request using the interface to the ratings service. If you re-add a rating that was removed, the original score will be restored. Automatic scores cannot be removed. Because qualification scores may represent private or proprietary information, the author of a score may declare it either public or private. Public qualification scores can be seen by anyone, and can be used by anyone as a qualification requirement. Private qualifications are only visible to the author and the user the score describes, and can only be used by the author as a qualification requirement.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: