Confidence-aware moderation for Laravel

Why probabilities, not labels

Thresholds, per category

The community preset flags harassment at 0.40 and blocks it at 0.95. Drag to see what a post's harassment probability earns. The marker is a real Jev answer: "anyone still recommending the Seiko 5 clearly has no idea what they are talking about", where fine was the top choice at 0.55.

Quick start

composer require phoenix1331/laravel-jev-moderation
php artisan jev-moderation:install

The install command publishes the config and migration, asks for the mode and default policy, and prints the .env keys. Add your key from typesafe.ai, then check it end to end:

JEV_API_KEY=...
php artisan jev-moderation:ping

Mark a model with the fields to send and a policy:

use Phoenix1331\JevModeration\Attributes\Moderate;
use Phoenix1331\JevModeration\Concerns\Moderatable;

#[Moderate(fields: ['title', 'body'], policy: 'community')]
class ForumPost extends Model
{
    use Moderatable;
}

Saving it records a pending decision and queues the check after the transaction commits. Run a queue worker, or set JEV_MODERATION_SYNC=true to check inline.

Laravel's default DatabaseSeeder uses WithoutModelEvents, which switches moderation off for every seeder it calls. Remove it from seeders whose content should be checked.

Using it

On the model

$post->moderationStatus();      // ModerationStatus, or null if never checked
$post->isFlagged();             // waiting for a human
$post->isBlocked();             // blocked, or rejected by a moderator
$post->isVisible();             // what the global scope would decide
$post->latestModeration;        // the decision that sets the status
$post->moderationDecisions;     // every decision, oldest first

$post->flagForReview('reported by user', $reporter);
$post->approve($moderator, 'false positive');
$post->reject($moderator, 'confirmed spam');
$post->reject($moderator, 'harassment, not spam', category: 'harassment');  // reclassify: removed under the corrected category
$post->moderateNow();           // check again, even if unchanged

Queries

ForumPost::query()->get();                  // visible content only
ForumPost::withoutModeration()->get();      // everything
ForumPost::onlyFlagged()->get();            // the review queue
ForumPost::onlyBlocked()->get();            // blocked or rejected
ForumPost::whereModerationStatus(ModerationStatus::Pending, ModerationStatus::Failed)->get();

Overrides

public function moderatableContent(): array    // field name to value; default: the #[Moderate] fields
public function moderationContext(): ?string   // e.g. the thread title or previous messages
public function moderationPolicy(): string     // default: the attribute's policy, else JEV_MODERATION_POLICY
public function moderationAuthor(): ?Model     // default: the user relation

Ad-hoc checks

Check a chat message before broadcasting it, without recording anything:

use Phoenix1331\JevModeration\Facades\JevModeration;

$result = JevModeration::classify($message, 'chat', $previousMessage);

if ($result->blocked()) {
    return;
}

Events

Each carries the model and the decision, and serialises for queued listeners: ContentPending, ContentPassed, ContentFlagged, ContentBlocked, ContentReported, ReviewResolved, and ModerationFailed (also with the reason and whether it was retryable). Sanctions such as muting an author belong in your listeners.

Policies and presets

A policy is one Jev question with a fixed set of choices. Descriptions are sent to Jev and are the main accuracy lever; the safe choice goes last, because Jev leans slightly towards the first option.

'community' => [
    'question' => 'Classify this user-generated forum content in `content`. ...',
    'choices' => [
        'spam' => 'Advertising, link farming, scams or repeated promotional content',
        'harassment' => 'Insults, threats, intimidation or targeted abuse of a person',
        // hate, personal_data, off_topic ...
        'fine' => 'Acceptable content that breaks none of these rules, ...',
    ],
    'safe_choice' => 'fine',
    'actions' => [
        'spam' => ['review_above' => 0.50, 'block_above' => 0.90],
        'harassment' => ['review_above' => 0.40, 'block_above' => 0.95],
        'personal_data' => ['review_above' => 0.40],
        // ...
    ],
    'uncertain_below' => 0.30,
],

Each policy has a version, a hash of everything Jev sees and every threshold, stored on each decision. After a change, jev-moderation:rescan re-checks content.

Measured, lightly

The presets are checked against 53 synthetic labelled posts with composer test:accuracy, against jev-1.14.0:

"Acceptable" is not ideal but not wrong, such as a rude but fair post being flagged. The set is small and the wording was tuned on it, so treat it as a sanity check and measure on your own content. Jev is decisive on clear cases, so most clear-cut posts are blocked rather than flagged; set block_above to 1.0 where every removal needs a person.

Modes and visibility

A global scope hides content by its latest decision. fail applies only to checks that gave up after their retries, or on an error such as a bad key.

For busy tables, php artisan jev-moderation:add-column forum_posts generates a migration adding an indexed moderation_status column, backfilled and kept in sync, so the scope reads a column instead of joining decisions.

Commands

Recorded against live Jev.

jev-moderation:install

jev-moderation:ping

jev-moderation:stats

jev-moderation:rescan

Stats pairs each moderator decision with Jev's latest earlier result for the record: upheld or overturned when Jev flagged or blocked, missed or report dismissed when Jev passed.

Rescan skips content a moderator approved or rejected unless --status names it, includes content never checked, and takes --since, --chunk and --dry-run.

Testing your app

Swap in the fake for every check (saves, queued jobs and classify()), set its answers, and assert on what was sent:

use Phoenix1331\JevModeration\Data\ModerationRequest;
use Phoenix1331\JevModeration\Facades\JevModeration;

JevModeration::fake()->classifyAs('spam', 1.0, ['spam' => 1.0]);

$post = ForumPost::query()->create(['title' => 'Cheap watches', 'body' => 'Buy now']);

expect($post->isBlocked())->toBeTrue();
JevModeration::fake()->assertClassified(fn (ModerationRequest $request): bool => $request->policy === 'community');

Also classifyUsing(), failWith(), one-off push() and pushFailure(), assertNotClassified(), assertClassifiedTimes() and assertNothingClassified(). Set JEV_MODERATION_DRIVER=fake in phpunit.xml so tests never reach Jev.

Configuration

jev-latest is an alias that moves when TypeSafe ships a release. Once your thresholds are tuned, pin a version such as JEV_MODEL=jev-1.14.0; every decision records the version that answered.

Security and data

Family

composer require phoenix1331/filament-jev-moderation