Skip to content

EvalMedia

Quality evals for generative media

EvalMedia is an open-source evaluation framework for AI-generated media.

Model providers update weights behind the same endpoint, prompt changes regress outputs, and generated files expire before you can compare them.

Star on GitHubThe form is for occasional updates. No newsletter.
Example output from an EvalMedia run: five named checks, each with a score and a pass or fail verdict.
evalmedia run ./outputs --suite video
image.prompt_adherence0.94PASS
image.artifact_detection0.02PASS
image.subject_consistency0.91PASS
video.temporal_stability0.88PASS
video.subject_consistency0.62FAIL
4 passed1 failed

Why generative media needs evals

Providers change the model under you.

Image and video providers ship new weights behind the same endpoint. Your code doesn’t change, your prompts don’t change, and your output quality changes anyway — silently, with nothing in your stack to flag it.

Prompt changes regress outputs.

An edit that fixes one case quietly breaks three others. Without a scored, repeatable check suite, “better” is a judgement call made on whichever sample you happened to look at.

Your generation history expires.

fal.ai deletes generated media after roughly 7 days and request payloads after 30. By the time you want to compare last month’s outputs against this month’s, the evidence is already gone.

What’s in the package

Open source, on PyPI. Install it and start scoring outputs.

Open Source

On PyPI

pip install evalmedia

  • Image checks

    Prompt adherence, artifact detection, and character and subject consistency.

    image.prompt_adherenceimage.artifact_detectionimage.subject_consistency

  • Video checks

    Temporal stability across frames, plus subject consistency over a clip.

    video.temporal_stabilityvideo.subject_consistency

  • fal.ai capture wrapper

    Wrap your fal calls and record prompts, parameters, and outputs at generation time.

  • CI and pytest

    Assert on check scores inside your test suite so quality regressions fail the build.

Frequently asked questions

What is EvalMedia?

EvalMedia is an open-source evaluation framework for AI-generated media. It runs quality checks on generated images and video, covering prompt adherence, artifact detection, character and subject consistency, and temporal stability. The Python package is published on PyPI as evalmedia.

How do you evaluate AI-generated image and video quality?

EvalMedia scores each output against a set of named checks rather than a single overall rating. Images are checked for prompt adherence, visual artifacts, and subject consistency; video adds temporal stability across frames. Every check returns a score you can assert on in pytest, so quality regressions fail CI instead of reaching users.

How do I know if a new model or prompt made my outputs worse?

Run a fixed set of prompts through EvalMedia before and after the change and compare the scores. Because providers ship new weights behind the same endpoint, a stable prompt set scored on the same checks is the only reliable way to detect drift.

Why do my fal.ai generations disappear?

fal.ai deletes generated media files after roughly 7 days and request payloads after 30 days. Once those windows pass, both the outputs and the inputs that produced them are gone, so you cannot re-score old runs. EvalMedia’s fal capture wrapper records generations as they happen, so you keep your own history.

Is EvalMedia open source?

Yes. The evaluation framework is open source and installable from PyPI as evalmedia.

Want to hear more?

Leave your email and we’ll write when there’s something new. No newsletter.