Plugin Development

AI AutoEvals provides a plugin system for creating custom fact extractors. This guide shows you how to create your own plugins.

Overview

Fact extractor plugins determine how evaluation criteria are extracted from user input. The module includes several built-in plugins:

• AI Generated: Uses LLM to extract facts
• Keyword: Keyword-based extraction
• Regex: Regex pattern-based extraction
• Hybrid: Combines multiple methods

You can create custom plugins for specialized fact extraction needs.

Plugin Architecture

Plugin Manager

Plugin Manager: plugin.manager.ai_autoevals.fact_extractor

Base Class

Base Class: Drupal\ai_autoevals\Plugin\FactExtractor\FactExtractorPluginBase

Interface

Interface: Drupal\ai_autoevals\Plugin\FactExtractor\FactExtractorPluginInterface

Creating a Custom Plugin

Step 1: Create Plugin Class

Create a PHP file in your module’s Plugin/FactExtractor directory.

<?php

namespace Drupal\my_module\Plugin\FactExtractor;

use Drupal\ai_autoevals\Plugin\FactExtractor\FactExtractorPluginBase;
use Drupal\Core\StringTranslation\StringTranslationTrait;

/**
 * Custom fact extractor plugin.
 *
 * @FactExtractor(
 *   id = "domain_specific",
 *   label = @Translation("Domain-Specific Extractor"),
 *   description = @Translation("Extracts facts for a specific domain."),
 *   weight = 10
 * )
 */
class DomainSpecificFactExtractor extends FactExtractorPluginBase {

  use StringTranslationTrait;

  /**
   * {@inheritdoc}
   */
  public function extract(string $input, array $context = []): array {
    $facts = [];

    // Extract technical requirements
    if (preg_match('/requires? (.+?)(?:\.|,|$)/i', $input, $matches)) {
      $facts[] = "The answer should include: " . trim($matches[1]);
    }

    // Extract numerical values
    if (preg_match_all('/\b\d+(?:\.\d+)?\s*(?:%|percent|seconds|minutes|hours|days|months|years)\b/i', $input, $matches)) {
      foreach ($matches[0] as $value) {
        $facts[] = "The answer should accurately reference: " . $value;
      }
    }

    // Extract dates
    if (preg_match_all('/\b\d{4}-\d{2}-\d{2}\b/', $input, $matches)) {
      foreach ($matches[0] as $date) {
        $facts[] = "The answer should include the date: " . $date;
      }
    }

    return $facts;
  }

  /**
   * {@inheritdoc}
   */
  public function isAvailable(): bool {
    // Only available if certain conditions are met
    return TRUE;
  }

}

Step 2: Plugin Annotation

The plugin annotation defines metadata about your plugin:

/**
 * @FactExtractor(
 *   id = "domain_specific",              // Unique identifier
 *   label = @Translation("..."),          // Display name
 *   description = @Translation("..."),   // Description
 *   weight = 10                          // Sort order
 * )
 */

Annotation Properties:

• id (required): Unique plugin identifier
• label (required): Human-readable name
• description (optional): Description of what the plugin does
• weight (optional): Sort order (lower numbers appear first)

Step 3: Required Methods

Your plugin must implement the following methods:

extract(string $input, array $context = []): array

Extract facts from the input.

public function extract(string $input, array $context = []): array {
  $facts = [];

  // Your extraction logic here

  return $facts;
}

Parameters:

• $input (string): User’s input/question
• $context (array): Additional context (previous turns, etc.)

Returns: array - Array of extracted facts (strings)

isAvailable(): bool

Check if the plugin is available for use.

public function isAvailable(): bool {
  // Check if required conditions are met
  return TRUE;
}

Returns: bool - TRUE if available, FALSE otherwise

Step 4: Clear Cache

Clear the plugin cache to discover your new plugin:

drush cache:rebuild

Plugin Selection

The fact extractor service selects the appropriate plugin based on the evaluation set configuration.

Using AI Generated Method

When the evaluation set uses “AI Generated” fact extraction, the AiFactExtractor plugin is used:

// The AI extractor uses custom knowledge if available
$facts = $factExtractor->extractFacts(
  $input,
  $context,
  $evaluationSet  // Contains custom knowledge
);

Using Hybrid Method

The hybrid extractor combines multiple methods:

// Hybrid extractor uses both AI and rule-based methods
$facts = $factExtractor->extractFacts(
  $input,
  $context,
  $evaluationSet
);

Advanced Plugin Examples

Plugin with Configuration

Create a plugin with custom configuration:

<?php

namespace Drupal\my_module\Plugin\FactExtractor;

use Drupal\ai_autoevals\Plugin\FactExtractor\FactExtractorPluginBase;
use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\Plugin\PluginFormInterface;

/**
 * Configurable fact extractor plugin.
 *
 * @FactExtractor(
 *   id = "configurable_extractor",
 *   label = @Translation("Configurable Extractor"),
 *   description = @Translation("Extracts facts based on custom patterns."),
 *   weight = 20
 * )
 */
class ConfigurableExtractor extends FactExtractorPluginBase implements PluginFormInterface {

  /**
   * {@inheritdoc}
   */
  public function extract(string $input, array $context = []): array {
    $facts = [];
    $patterns = $this->getConfiguration()['patterns'] ?? [];

    foreach ($patterns as $pattern) {
      if (preg_match($pattern['regex'], $input, $matches)) {
        $facts[] = str_replace('{match}', $matches[1] ?? '', $pattern['fact_template']);
      }
    }

    return $facts;
  }

  /**
   * {@inheritdoc}
   */
  public function buildConfigurationForm(array $form, FormStateInterface $form_state): array {
    $form['patterns'] = [
      '#type' => 'textarea',
      '#title' => $this->t('Patterns (JSON)'),
      '#default_value' => json_encode($this->getConfiguration()['patterns'] ?? [], JSON_PRETTY_PRINT),
      '#description' => $this->t('JSON array of patterns with regex and fact_template.'),
    ];

    return $form;
  }

  /**
   * {@inheritdoc}
   */
  public function submitConfigurationForm(array &$form, FormStateInterface $form_state): void {
    $patterns = json_decode($form_state->getValue('patterns'), TRUE);
    $this->setConfiguration(['patterns' => $patterns]);
  }

}

Plugin with Custom Knowledge

Access custom knowledge from the evaluation set:

public function extract(string $input, array $context = [], ?EvaluationSetInterface $evaluationSet = null): array {
  $facts = [];

  // Access custom knowledge
  if ($evaluationSet && $evaluationSet->hasCustomKnowledge()) {
    $knowledge = $evaluationSet->getCustomKnowledge();
    $facts[] = "The answer should be consistent with: " . substr($knowledge, 0, 100);
  }

  return $facts;
}

Plugin with Conversation Context

Use conversation context for better extraction:

public function extract(string $input, array $context = []): array {
  $facts = [];
  $previousTurns = $context['previous_turns'] ?? [];

  // Check if this is a follow-up question
  if (!empty($previousTurns)) {
    $lastInput = end($previousTurns)['input'] ?? '';
    $facts[] = "The answer should build upon previous context about: " . substr($lastInput, 0, 50);
  }

  return $facts;
}

Plugin Testing

Unit Testing

<?php

namespace Drupal\Tests\my_module\Unit;

use Drupal\Tests\UnitTestCase;
use Drupal\my_module\Plugin\FactExtractor\DomainSpecificFactExtractor;

class DomainSpecificFactExtractorTest extends UnitTestCase {

  protected $extractor;

  protected function setUp(): void {
    parent::setUp();
    $this->extractor = new DomainSpecificFactExtractor([], '', []);
  }

  public function testExtractTechnicalRequirements(): void {
    $input = 'What are the requirements for the API?';
    $facts = $this->extractor->extract($input);

    $this->assertContains('The answer should include: the API', $facts);
  }

  public function testExtractNumericalValues(): void {
    $input = 'What is the response time? It should be under 200ms.';
    $facts = $this->extractor->extract($input);

    $this->assertContains('The answer should accurately reference: 200ms', $facts);
  }

}

Kernel Testing

<?php

namespace Drupal\Tests\my_module\Kernel;

use Drupal\KernelTests\KernelTestBase;

class FactExtractorPluginTest extends KernelTestBase {

  protected static $modules = ['ai_autoevals', 'my_module'];

  public function testPluginIsDiscovered(): void {
    $plugin_manager = $this->container->get('plugin.manager.ai_autoevals.fact_extractor');
    $plugin = $plugin_manager->createInstance('domain_specific');

    $this->assertInstanceOf(DomainSpecificFactExtractor::class, $plugin);
  }

  public function testPluginExtraction(): void {
    $plugin_manager = $this->container->get('plugin.manager.ai_autoevals.fact_extractor');
    $plugin = $plugin_manager->createInstance('domain_specific');

    $facts = $plugin->extract('What is the timeout? 30 seconds.');

    $this->assertNotEmpty($facts);
  }

}

Best Practices

1. Keep Extractors Fast

Avoid slow operations in your extractors:

// Good - Simple regex
if (preg_match('/timeout.*?(\d+)/i', $input, $matches)) {
  $facts[] = "The answer should reference: " . $matches[1] . " seconds";
}

// Bad - External API calls
$response = file_get_contents('https://api.example.com/extract');

2. Handle Edge Cases

Handle various input formats:

public function extract(string $input, array $context = []): array {
  if (empty(trim($input))) {
    return [];
  }

  // Normalize input
  $input = trim($input);

  // Extract facts
  // ...
}

3. Provide Clear Facts

Generate clear, specific evaluation criteria:

// Good - Specific
$facts[] = "The answer should state that the timeout is 30 seconds";

// Bad - Vague
$facts[] = "The answer should mention timeout";

4. Use Context Wisely

Leverage conversation context when available:

public function extract(string $input, array $context = []): array {
  $facts = [];

  // Use previous turns for context
  if (!empty($context['previous_turns'])) {
    $facts[] = "The answer should be consistent with previous conversation";
  }

  // Extract from current input
  // ...

  return $facts;
}

5. Document Your Plugin

Add clear documentation:

/**
 * Product-specific fact extractor.
 *
 * This extractor generates evaluation criteria based on product knowledge.
 * It focuses on product specifications, features, and pricing information.
 *
 * @FactExtractor(
 *   id = "product_extractor",
 *   label = @Translation("Product Extractor"),
 *   description = @Translation("Extracts product-related facts for evaluation."),
 *   weight = 15
 * )
 */

Next Steps

• API Reference - Complete service documentation
• Event System - Event system guide
• Examples - Real-world implementations
• Extending the Module - Extension guide