Installation

This guide covers the installation process for AI AutoEvals.

Prerequisites

Before installing AI AutoEvals, ensure you have:

• Drupal 10.2+ or Drupal 11 installed and configured
• AI module (version 1.0+) - Download from Drupal.org
• Key module - Download from Drupal.org
• A configured AI provider - See AI module documentation for setup instructions

Installation Steps

1. Install via Composer using Composer:

   composer require drupal/ai_autoevals

2. Enable the Module using Drush:

   drush en ai_autoevals

   Or enable via the admin interface at /admin/modules

3. Configure the Module at /admin/config/ai/autoevals:

   Required Settings:

   • Default Provider: Select your AI provider from the dropdown
   • Default Model: Choose the model to use for evaluations
   • Operation Types: Select which AI operations to evaluate (chat, chat_completion)

   Optional Settings:

   • Auto-track: Automatically evaluate all matching requests
   • Fact Extraction Method: Choose evaluation criteria extraction strategy
   • Context Depth: Set conversation history depth (0-10 turns)
   • Retention Period: Days to keep evaluation results (0 = keep forever)
   • Global Exclusion Keywords: Keywords that prevent evaluation across all evaluation sets
     • Query Exclusion Keywords: Skip evaluation if any keyword appears in user’s query
     • Response Exclusion Keywords: Skip evaluation if any keyword appears in AI’s response
     • Global exclusions take priority over per-set exclusions and inclusion keywords

4. Configure Permissions by assigning appropriate permissions to user roles:

   Permission	Description
   administer ai autoevals	Full administration access
   view ai autoevals results	View evaluation results
   edit ai autoevals results	Edit and re-evaluate results
   delete ai autoevals results	Delete results
   view ai autoevals sets	View evaluation sets
   manage ai autoevals sets	Create/edit/delete evaluation sets
   requeue ai autoevals	Requeue failed evaluations
   batch ai autoevals	Perform batch operations

5. Verify Installation:

   • Visit the dashboard at /admin/content/ai-autoevals
   • You should see the dashboard with initial statistics
   • Make an AI request with auto-tracking enabled
   • Run the evaluation queue: drush queue:run ai_autoevals_evaluation_worker
   • Check the dashboard for new evaluations

Troubleshooting Installation

Module Not Appearing

• Clear all caches: drush cr
• Check that dependencies are enabled: drush pm:info ai key
• Verify Composer installation completed successfully

AI Provider Not Available

• Ensure AI module is properly configured
• Check that you have added API keys for your provider
• Visit /admin/config/ai/providers to verify provider setup

Permissions Not Showing

• Clear caches: drush cr
• Rebuild permissions: drush config:import (if needed)

Upgrading

1. Backup your site and database

2. Update via Composer: composer update drupal/ai_autoevals

3. Run database updates: drush updatedb

4. Clear caches: drush cr

5. Review the release notes for breaking changes

Uninstallation

1. Disable the module: drush pm:uninstall ai_autoevals

2. Remove from Composer: composer remove drupal/ai_autoevals

Note: Uninstalling will delete all evaluation results and evaluation sets. Export any important data before uninstalling.