Commits

littledot5566  committed 8de3ece

Updated latex.

  • Participants
  • Parent commits fcff7b1

Comments (0)

Files changed (13)

File latex/BracketScore.png

Added
New image

File latex/BracketTasks.png

Added
New image

File latex/CS886PAPER.pdf

Binary file modified.

File latex/CS886PAPER.tex

 
 The requester sets a base pay ($B$), a maximum ($H$) and minimum ($L$) amount of pay a worker can be rewarded for completing a task. When a newcomer enters the system, he is rewarded the base pay. Once each worker is assigned a score, LadderMT sorts the workers in decreasing order, generating a ranking. Their pay is then adjusted by payment rules.
 
-\subsection{Linear Payout Rule}
+\subsection{Linear Payment Rule (LPR)}
 
 The linear payout rule rewards the highest ranking worker with the highest pay set and decreases linearly with regard to the worker's rank, with the lowest ranking worker rewarded with the lowest pay set.
 
-Let $i$ denote a worker's rank, $w$ denote the total number of workers in the ranking system; then the worker's pay ($P$) could be formulated as:
+Let $i_w$ denote a worker's rank, $w$ denote the total number of workers in the ranking system; then the worker's pay ($P$) could be formulated as:
+\[ P = L + (H-L)\cdot((w-i_w)/w) \]
 
-\[ P = L + (H-L)\cdot((w-i)/w) \]
-
-\subsection{Bracket Payout Rule}
+\subsection{Bracket Payment Rule (BPR)}
 
 The bracket payout rule splits the ladder into partially ordered groups by flooring workers' scores and grouping workers with the same floored score in one bracket. Similar to the linear payout rule, the bracket payout rule rewards the highest ranking bracket with the highest pay set and decreases linearly with regard to the number of brackets, with the lowest ranking bracket rewarded with the lowest pay set. Each member in the group receives the same pay.
 
+For example, if a worker has a raw score of 0.58, after flooring his score, he will be assigned to the bracket of score 0, the lowest bracket in the system. In contrast, if a worker has the highest possible score of 5.5, he will be assigned to the highest bracket of score 5 after flooring his raw score.
+
+Workers are paid based on their bracket when completing tasks. The higher the bracket a worker is in, the more money the worker will be awarded for completing tasks. Let $b$ denote the total number of brackets, $i_b$ denote the rank of the bracket; the pay difference $\Delta_P$ between brackets and worker pay can be formulated as
+\[ \Delta_P = (H - L) / b \]
+\[ P = L + (b-i_b) \cdot \Delta_P \]
+
+For example, suppose $(L,H,b)=(20,60,6)$, the interval payment will be 8 after applying the formula above. Therefore, workers will be paid 20, 28, 36, 44, 52, and 60 depending on the bracket he is in. 
+
+
 \section{Simulation}
 
 We simulate our model by generating $w$ workers to complete $t$ tasks. The same set of workers is used in a fixed pay environment and our LadderMT environment.
 Each worker is initialized with a randomly generated expected quality $e(Q) \in \{x \in \mathbb{R} \mid 0 \leq x \leq 1\}$ and standard deviation $\sigma \in \{x \in \mathbb{R} \mid 0 \leq x\leq 1\}$. We assume that workers' output quality should conform to a normal distribution. Thus, we create a Gaussian random number generator for each worker with his expected quality as mean and standard deviation as is to generate work qualities $Q \in \{x \in \mathbb{R} \mid 0 \leq x\leq 1\}$.
 
 Given a task, workers decide whether or not to participate in completing it. We call the probability that a worker will participate in completing a task the \textit{attractiveness} of the task. Let $\widebar{P}$ denote the entire market's average task pay; we model a task's attractiveness ($A$) as:
-
 \[ A = P^2 / (2 \cdot \widebar{P}^2) \]
 
 When $A \geq 1$, workers will repeatedly work on the tasks; when $A = 0$, workers will find the task unattractive and find other activities to do. When a task pays the same amount as the market average $(P = \widebar{P})$, there is equal chance that a worker will participate in completing a task or other activities. Since workers aim to maximize their utility, we assume that workers who choose to opt out of our tasks found some other task in the market that pays higher for the same amount of work; whatever other tasks workers engage in, it will always take the same amount of time as completing our task; thus all workers start on some task at the same time, work in parallel and complete their tasks at the same time. Then again, workers come back to decide on what tasks to engage in.
 
 \section{Performance Evaluation}
 
-We evaluate our model based on its effectiveness to attract top-tier workers to return and complete more tasks while repelling bottom-tier workers, the total amount of capital spent and the overall quality of completed tasks. We simulate our model using 5 workers completing 100 tasks.
+We evaluate our model based on its effectiveness to attract top-tier workers to return and complete more tasks while repelling bottom-tier workers, the total amount of capital spent and the overall quality of completed tasks. For simplicity, we run a small-world simulation using 5 workers completing a total of 100 tasks; values regarding payment are set to $(\widebar{P}, B, L, H)=(50, 50, 20, 60)$. We set their expected quality by hand. 
 
-\subsection{Methodology 1 - Performance Based Rating System}
+\subsection{Fixed Pay Environment}
 
 %The Performance Based Rating System (PBRS) is to pay workers base on their score. A worker who has the highest score will receive the maximum pay, and the lowest score will receive the lowest pay. We sort workers base on their score begin from the highest score, so for each worker i, we can have following payout rule:
 %Payment = (Maximum Pay – Minimum Pay) * ((No. Workers – i)  /  No. Workers)
 
 
+We show results of these five workers working in a fixed pay environment. The pay is fixed to the base pay. Figure \ref{fig:FIG1} shows workers' $E(Q)$ and $\widebar{Q}$. We manually set $E(Q)$ so that our workers are of diverse quality. Figure \ref{fig:FIG2} shows the task distribution among workers. Because the pay is fixed and is the same as the market average, there are no incentives for high-quality workers to do more tasks, they may do other requesters' tasks. In contrast, low-quality workers are equally incentivised as high-quality workers, taking up similar proportions of the task quota. After running 10 simulations, we obtain a mean average quality of 0.5094\% with deviation of 0.0181 spending a mean budget of \$5,000 with a deviation of 0.
+
+\begin{figure}[h]
+\centering
+\includegraphics[width=\linewidth]{FIG1}
+\caption{Workers expected work quality $E(Q)$ and average work quality $\widebar{Q}$ of completed tasks. $E(Q)$ is manually set for this experiment.}
+\label{fig:FIG1}
+\end{figure}
+
+\begin{figure}[h]
+\centering
+\includegraphics[width=\linewidth]{FIG2}
+\caption{Task distribution among workers in a fixed pay environment. Bar graph represents amount paid for completing tasks, corresponding to the axis on the left; line graph represents the number of tasks completed, corresponding to the axis on the right.}
+\label{fig:FIG2}
+\end{figure}
+
+\subsection{LadderMT using Linear Payout Rule}
+
+We show results of these five same workers working in LadderMT, a dynamic pay environment using the Linear Payout Rule. Recall that after completing tasks, LadderMT first calculate each worker's score, then adjust their pay for future tasks based on their score. Figure \ref{fig:FIG4} shows each worker's score after completing a task. LadderMT assigns a higher score to Worker 0 because of his high-quality submissions. This results in a higher rank for Worker 0, netting him higher pay for future tasks. Figure \ref{fig:FIG3} shows the task distribution among workers. Worker 0 completes much more tasks than all other workers because he gets paid more, making the task much more attractive to him, which results in a higher probability of him working on our tasks rather than other requesters' tasks every round. In contrast, Worker 4 has the lowest score, netting him a pay much lower than the market average; at this point, it makes sense for him to find other tasks on the market to do which pays more than our tasks. This is also why Worker 0's line is longer than Worker 4's line in Figure \ref{fig:FIG4}. After running 10 simulations, we obtain a mean average quality of 64.06\% with deviation of 2.5\% spending a mean budget of \$4715.20 with a deviation of \$106.36
+
+
+\begin{figure}[h]
+\centering
+\includegraphics[width=\linewidth]{FIG4}
+\caption{LadderMT's worker scoring rule correctly catalogs high-quality workers with higher scores than low-quality workers.}
+\label{fig:FIG4}
+\end{figure}
+
+
+\begin{figure}[h]
+\centering
+\includegraphics[width=\linewidth]{FIG3}
+\caption{Task distribution among workers in LadderMT, under the Linear Payout Rule.}
+\label{fig:FIG3}
+\end{figure}
+
+
+
+\subsection{LadderMT using Bracket Payout Rule}
+
+%Our second approach, is to make our rating system floored, unlike previous approach; this allows our system to eliminate many low quality workers. In FSRS, we divided our system to 6 floors (rating 0 to rating 5), and the worker's score is floored whenever needed.
+
+We show results of these five same workers working in LadderMT, a dynamic pay environment using the Bracket Payout Rule. Figure \ref{fig:BS}, all workers start from the lowest bracket. We can see Worker 0 outperforms all other workers, since Worker 0 is a highest-quality worker; whereas Worker 4, the lowest-quality Worker, stays in the lowest bracket. Figure 7 also shows that high-quality workers are more likely to complete tasks rather than low-quality workers.  This is because low quality workers, who stay in the rank 0, will receive the lowest pay, which makes the task less attractive to them. After running 10 simulations, we obtain a mean average quality of 66.96\% with deviation of 2.08\% spending a mean budget of \$4067.20 with a deviation of \$103.3793.
+
+We believe Bracket Payment Rule is better than Linear Payment Rule because BPR only provide higher payment when the worker is actually high-quality worker. Each worker in LPR cannot easily climb to the next level if it only produces some quality works; instead, workers need to output certain amounts of high-quality works in order to get to the next level.
+
+\begin{figure}[h]
+\centering
+\includegraphics[width=\linewidth]{BracketScore}
+\caption{Bracket Payout Rule floors the score of each worker and group them into brackets of the same score.}
+\label{fig:BS}
+\end{figure}
+
+\begin{figure}[h]
+\centering
+\includegraphics[width=\linewidth]{BracketTasks}
+\caption{Task distribution among workers, under the Bracket Payout Rule}
+\label{fig:BT}
+\end{figure}
+
+\subsection{Introducing Malicious Workers}
+
+In this section, we focus on malicious workers. We assume that requesters always accept submissions from workers. We define a malicious worker as a worker that will produce low quality results when he/she performs numerous repeated tasks in a bracket. We employ the same workers as above, except they are now malicious. Figure \ref{fig:M1} presents the average qualities and expected average qualities of malicious workers. Since the workers are malicious, most of them have lower average qualities compared to expected average qualities. 
+
+\begin{figure}[h]
+\centering
+\includegraphics[width=\linewidth]{M1}
+\caption{Task distribution among workers, under the Bracket Payout Rule}
+\label{fig:M1}
+\end{figure}
+
+\begin{figure}[h]
+\centering
+\includegraphics[width=\linewidth]{M2}
+\caption{Task distribution among workers, under the Bracket Payout Rule}
+\label{fig:M2}
+\end{figure}
+
+\begin{figure}[h]
+\centering
+\includegraphics[width=\linewidth]{M3}
+\caption{Task distribution among workers, under the Bracket Payout Rule}
+\label{fig:M3}
+\end{figure}
+
+\begin{figure}[h]
+\centering
+\includegraphics[width=\linewidth]{M4}
+\caption{Task distribution among workers, under the Bracket Payout Rule}
+\label{fig:M4}
+\end{figure}
+
+\begin{figure}[h]
+\centering
+\includegraphics[width=\linewidth]{M5}
+\caption{Task distribution among workers, under the Bracket Payout Rule}
+\label{fig:M5}
+\end{figure}
+
+Since we construct the scoring system with the historical quality of works, the lower quality works will result to the lower payments. Therefore we assumed that the workers will pay more effort to contribute higher quality works, when his/her payments reached to a certain low level. First we performed the simulation with 100 tasks and 5 workers.
+
 
-We simulate the result by generating 5 workers (agents). In figure 1, we can see the behavior for each agent. For example, agent 0 is the high-quality agent, whereas the agent 4 is the low quality agent. We ask these 5 agents to do our 100 static tasks with \$50 of the paid; the result is shown in Figure 2.  Since all agents are being paid with same amount of money no matter how high the quality they have produced, each of them will always have equal probability to complete the job.
 
-If we put the same agents into our LadderMT simulation with the pay set between \$20 and \$60, we can achieve better average quality and less total payout (see figure 3). The average quality in static task was only 0.53 and we need to pay totally \$5000 for completing 100 tasks. However, in PBRS, we can have 0.64 is average quality but only requires \$4748 payment. Capital reduction is approximately 5.04\%. This is because the higher quality agents are more likely to complete tasks because we pay them more. On the other hand, the low quality agents are not interested in our task because they don't get paid well (see figure 4). For example, in figure 5, agent 0 will stabilize around a score of 5, and complete most of the tasks because the pay is good; in contrast, agent 4's score is always low, unable to reach a score of 0.5, he is also discouraged to complete hardly any tasks at all.
+Figure \ref{fig:M2} shows the qualities of tasks with fixed payments. After the malicious workers completed some tasks, they starts to produce low quality works in fixed payment environment. Figure \ref{fig:M3} represents the qualities of the workers with their completed tasks. It shows that the qualities of malicious workers are not stable. 
 
 
 
-\subsection{Methodology 2 - Floored Style Rating System}
+Figure \ref{fig:M4} shows the scores with the tasks. The qualities are positively correlated to the scores. Figure \ref{fig:M5} illustrates the payments of each task. The payments are based quality. For example, the quality of Worker 0 dropped to zero at its 12th task. Its payments reduced from 60 to 44. We assumed that every worker expects a higher payment. Therefore Worker 0 contributed higher quality work after its 18th task. LadderMT drives the malicious workers to contribute more higher quality works. Then we performed 10 repetitions that simulated with 10,000 tasks and 50 malicious workers.  In the fixed payment environment, the average quality is only 2.3\%. With LadderMT, the average quality is 55\%. The average total payment is reduced 4.57\% with LadderMT payment scheme.
+ 
+\section{Conclusion}
 
-//這段用上面取代Our second approach, Floored Style Rating System (FSRS), is to make our rating system floored, unlike previous approach; this allows our system to eliminate many low quality workers. In FSRS, we divided our system to 6 floors (rating 0 to rating 5), and the worker's score is floored whenever needed.
-	For example, if a worker has a raw score of 0.58, after flooring his score, he will be assigned to the bracket of score 0, the lowest bracket in the system. In contrast, if a worker has the highest possible score of 5.5, he will be assigned to the highest bracket of score 5 after flooring his raw score. Hence, for our simulation, we have 6 brackets.
-	Workers will be paid based on their bracket when completing tasks. This means that the higher the bracket a worker is in, the more money the worker will be awarded for completed tasks. The interval pay and worker's task pay is calculated based on the following formula.
-Interval payment = (Maximum Pay – Minimum Pay) / Num of intervals
-Take our previous payment setting as an example, with minimum pay of 20 and maximum pay of 60 and base pay of 50, the interval payment will be 8 after applying the formula above. Therefore, our paying rule P for this setting will be 20, 28, 36, 44, 52, and 60 for the six brackets. 
-P = Low + (interval Rank * Interval payment)
 
-The figure 6 – figure 8 is the result of FSRS approach. In figure 6, all agents start from the lowest bracket. We can see the agent 0 outperforms all other agents, since agent 0 is a highest-quality worker; whereas agent 4, the lowest-quality worker, stays in the lowest bracket. Figure 7 also shows that high-quality workers are more likely to complete tasks rather than low-quality workers.  This is because low quality workers, who stay in the rank 0, will receive the lowest pay, which makes the task less attractive to them.
-	We believe FSRS is better than PBRS because FSRS only provide higher payment when the agent is actually high-quality agent. Each agent in FSRS cannot easily climb to the next level if it only produces some quality works; instead, all agents need to output certain amounts of high-quality works in order to get to the next level. In figure 8, we show that the total pay of the FSRS only requires \$3892, and the reduce ratio is approximately 0.8, which is better than the FSRS' 0.5.
 
 
 

File latex/FIG1.png

Added
New image

File latex/FIG2.png

Added
New image

File latex/FIG3.png

Added
New image

File latex/FIG4.png

Added
New image

File latex/M1.png

Added
New image

File latex/M2.png

Added
New image

File latex/M3.png

Added
New image

File latex/M4.png

Added
New image

File latex/M5.png

Added
New image