Troubleshooting

This guide covers common issues and their solutions when using AI AutoEvals.

Evaluations Stuck in “Pending”

Problem

Evaluations remain in “pending” status and don’t get processed.

Solutions

1. Process the Queue Manually

drush queue:run ai_autoevals_evaluation_worker

2. Check Cron Configuration

Ensure cron is running regularly:

drush cron:run

Configure cron to run at least hourly for best results.

3. Check Queue System

Verify the queue is not blocked:

drush queue:info

Look for the ai_autoevals_evaluation_worker queue and check its size.

4. Review System Resources

Check if the server has enough resources:

• Memory
• CPU
• Disk space

5. Check Locks

Clear any stuck locks:

drush lock:break

No Evaluations Being Created

Problem

AI requests are not being tracked and no evaluations are created.

Solutions

1. Verify Auto-Track is Enabled

1. Go to /admin/config/ai/autoevals
2. Check that “Auto-track requests” is enabled
3. Save the configuration

2. Check Operation Types

Ensure the operation types match your AI requests:

• chat
• chat_completion

3. Verify AI Provider Configuration

1. Visit /admin/config/ai/providers
2. Confirm your provider is properly configured
3. Test the provider connection

4. Check Event Subscription

Ensure the module’s event subscriber is registered:

drush cache:rebuild

5. Review Drupal Logs

Check for errors in the watchdog log:

drush watchdog:show --filter="ai_autoevals" --full

Or visit /admin/reports/dblog and filter for “ai_autoevals”.

6. Verify Permissions

Ensure the user making AI requests has proper permissions:

• View error messages in logs for permission errors
• Check user roles and permissions

Low Success Rate

Problem

Many evaluations are failing or returning unexpected results.

Solutions

1. Check API Quota

Verify your AI provider has sufficient API quota:

# Check rate limits for your provider
# OpenAI: https://platform.openai.com/account/limits
# Anthropic: https://console.anthropic.com/settings/limits

2. Verify Model Availability

Ensure the evaluation model is available and accessible:

# Test the model directly
drush ai:test --provider=openai --model=gpt-4

3. Review Error Logs

Check detailed error logs:

drush watchdog:show --filter="ai_autoevals" --severity=error --full

4. Check Timeout Settings

Evaluations may timeout on slow responses:

• Reduce max_tokens in the evaluation prompt
• Use a faster model for evaluations
• Increase PHP timeout settings

5. Verify Response Format

Ensure the evaluation model returns responses in the expected format:

• Choice (A, B, C, D)
• Analysis text

Some models may need additional prompting to follow the format.

6. Test with Simple Input

Create a manual evaluation with simple input to isolate the issue:

$evaluation = \Drupal::service('ai_autoevals.evaluation_manager')->createEvaluation([
  'evaluation_set_id' => 'default',
  'request_id' => 'test-123',
  'provider_id' => 'openai',
  'model_id' => 'gpt-4',
  'operation_type' => 'chat',
  'input' => 'What is 2+2?',
  'output' => '4',
]);

Queue and process this evaluation to test the basic functionality.

Incorrect Scores

Problem

Evaluations are returning unexpected or incorrect scores.

Solutions

1. Review Evaluation Set Configuration

Check the evaluation set configuration:

• Verify choice scores are set correctly
• Review custom prompt templates
• Check fact extraction method

2. Examine Extracted Facts

View individual evaluations to see what facts were extracted:

• Navigate to /admin/content/ai-autoevals/results
• Click on an evaluation
• Review the “Facts” section

If facts are incorrect, adjust the fact extraction method or add custom knowledge.

3. Test Different Evaluation Sets

Create different evaluation sets with varying configurations:

• Adjust scoring weights
• Modify prompts
• Try different fact extraction methods

4. Compare Against Human Judgment

Evaluate a sample of responses manually:

• Compare AI scores with human judgment
• Identify patterns in discrepancies
• Adjust configuration to match human judgment

5. Review Custom Knowledge

If using custom knowledge, ensure it’s:

• Accurate and up-to-date
• Relevant to the questions being evaluated
• Clear and well-structured

Performance Issues

Problem

Evaluations are slow or causing performance issues.

Solutions

1. Process Queue During Off-Peak Hours

Schedule queue processing for low-traffic times using cron:

# In crontab
0 2 * * * drush queue:run ai_autoevals_evaluation_worker

2. Use Faster Evaluation Model

Switch to a faster model for evaluations:

• GPT-3.5 instead of GPT-4
• Claude Haiku instead of Claude Opus

3. Reduce Context Depth

Lower the context depth setting:

• Go to /admin/config/ai/autoevals or edit evaluation set
• Reduce “Context Depth” from 10 to 3 or less
• Less context means faster evaluations

4. Limit Concurrent Evaluations

The queue processes one evaluation at a time by default. This prevents overwhelming the system.

5. Enable Caching

Fact extraction uses caching. Verify it’s working:

drush cache:info | grep ai_autoevals_facts

Clear caches if needed:

drush cache:rebuild

6. Optimize Batch Operations

When performing batch re-evaluations:

• Process in smaller batches (50-100 at a time)
• Process during off-peak hours
• Monitor system resources

Database Issues

Problem

Evaluations are not being saved or retrieved correctly.

Solutions

1. Check Database Tables

Verify the database tables exist:

drush sql:query "SHOW TABLES LIKE '%ai_autoevals%'"

You should see:

• ai_autoevals_evaluation_result
• ai_autoevals_evaluation_set
• ai_autoevals_evaluation_result_revision (if versioning is enabled)

2. Run Database Updates

drush updatedb

3. Reinstall the Module

As a last resort, reinstall:

drush pm:uninstall ai_autoevals
drush en ai_autoevals

Warning: This will delete all evaluation data.

4. Check Database Permissions

Ensure Drupal has proper database permissions:

• SELECT
• INSERT
• UPDATE
• DELETE
• CREATE
• ALTER

Module Not Working After Update

Problem

After updating AI AutoEvals, the module is not functioning correctly.

Solutions

1. Run Database Updates

drush updatedb

2. Clear All Caches

drush cache:rebuild

3. Review Release Notes

Check the release notes for:

• Breaking changes
• Required configuration updates
• New dependencies

4. Check Dependencies

Verify all dependencies are up to date:

composer outdated drupal/ai
composer update drupal/ai

5. Rebuild Services

drush cache:rebuild
drush cr

Getting Help

If you’re still experiencing issues:

1. Check the logs: Always check the watchdog log first
2. Review configuration: Verify all settings are correct
3. Test with simple cases: Create simple test evaluations
4. Search the issue queue: Drupal.org issue queue
5. Report a bug: Create an issue with:
   • Drupal version
   • AI module version
   • PHP version
   • Steps to reproduce
   • Expected vs actual behavior
   • Any error messages
   • Relevant log entries

Next Steps

• Learn about Evaluation Sets
• Read the API Reference
• View Examples