AI Agent Evaluator

AI Agent Evaluator

Employment Type: Contractor
Work Arrangement: Remote
Eligible Location: United States
Education: Bachelor's degree required
Working Environment: Smartphone-based
Project Kickoff: October 12

Compensation

Hourly Rate: $25–$45 per hour

Estimated Time per Task: Approximately 1.5 hours per run, including evidence collection and scoring.

Actual earnings depend on task availability, completed assignments, and hours worked.

About the Role

We are seeking detail-oriented, US-based AI Agent Evaluators to test and assess how effectively AI agents complete everyday tasks on behalf of users.

You will test two AI agents side by side using your own accounts and smartphone. Tasks may involve everyday activities such as shopping, booking travel, managing email follow-ups, completing forms, and organizing personal logistics.

Your role is to evaluate how well each agent performs, determine whether it follows instructions, monitor important actions, and document the experience through screen recordings and structured assessments.

This opportunity is suited to people who already use AI tools in their daily lives and can provide clear, objective feedback. No coding or specialized AI background is required.

Key Responsibilities

AI Agent Testing

• Test two AI agents across realistic, everyday user scenarios.

• Execute tasks involving bill payments, returns, email follow-ups, bookings, shopping, groceries, travel, family logistics, outings, and forms.

• Observe how agents interpret instructions and complete multi-step tasks.

• Approve or stop agent actions at designated decision points.

• Evaluate whether agents complete tasks accurately and deliver the intended results.

Screen Recording & Evidence Collection

• Screen record every testing attempt using a smartphone.

• Capture the agent's actions and relevant steps throughout each task.

• Ensure recordings provide sufficient evidence to assess the agent's performance.

• Document issues, unexpected behavior, incomplete actions, and other relevant observations.

Performance Scoring & Feedback

• Score each run based on task completion, output quality, control issues, user satisfaction, and time taken.

• Compare the performance of both agents using consistent evaluation criteria.

• Provide clear written feedback about what worked, what failed, and how the experience could improve.

• Follow project instructions and documentation requirements carefully.

• Deliver accurate, detailed assessments that support the improvement of AI agent capabilities.

Required Qualifications

• Must be based in the United States.

• Bachelor's degree required.

• Two to three years of professional experience, preferably in engineering, business management, office administration, or operations.

• Regular personal use of AI tools in everyday life.

• Active personal account with a supported AI service, using the Free, Go, or Plus plan; Pro accounts are not eligible.

• Personal smartphone capable of screen recording.

• Strong attention to detail and ability to follow instructions precisely.

• Clear written communication skills.

• Ability to evaluate task outcomes objectively and document findings accurately.

Preferred Qualifications

• Experience using AI assistants or AI-powered productivity tools.

• Background in business operations, engineering, administration, or management.

• Experience testing software, evaluating user experiences, or documenting task workflows.

• Ability to identify errors, inconsistencies, and points where automated actions require user intervention.

Generalists are welcome, while relevant domain expertise is an advantage.

Key Skills

• ChatGPT

• Muse

• AI Agents

• AI Agent Evaluation

• Task Completion Assessment

• Instruction Following

• Quality Scoring

• Smartphone Screen Recording

• Evidence Collection

• User Experience Evaluation

• Attention to Detail

• Written Communication

• Process Documentation

• Critical Thinking

• Following Evaluation Guidelines

Work Environment

This is a remote, contractor-based opportunity involving smartphone-based testing. You can work on your schedule, subject to project requirements and task availability.

Each testing run takes approximately 1.5 hours, including evidence collection and scoring. You will use your personal accounts and smartphone to evaluate AI agents performing realistic everyday tasks.

Selection Process

1. Submit an application.

2. Complete the screening questions.

3. Attend a 15–30-minute interview.

4. Undergo profile review and client presentation.

5. Complete onboarding with the project operations team.

What Success Looks Like

Success means producing thorough, consistent, and evidence-based evaluations of AI agents. Your work should accurately capture task completion, output quality, control issues, user satisfaction, and execution time while identifying opportunities to improve the overall user experience.