Articles Snippets Projects

The Flaky Test Chronicles VI: The Reference

The Complete Checklist and AI Prompt for Test Reliability

January 1st ʼ26 2 months ago 8 min 1520 words

Five parts. 300+ commits. 43 alias mocks eliminated. One mission: tests that pass every time.

You've made it through the chronicles. The Reckoning taught you why tests fail randomly. Mock Madness showed you the alias mock trap. The Determinism Principle gave you control over time and randomness. The Teardown Tango mastered cleanup. The Abstraction Avalanche tamed your TestCase architecture.

Now you need a reference. Something to pull up during code review. Something to hand to a new team member. Something to feed your AI assistant.

This is that reference.


The Ultimate Checklist

Everything from Parts 1-5, organized by category.

Time & Randomness

  • Freeze time at test start with Carbon::setTestNow() or travelTo()

  • Use travelBack() after travelTo(), or use the callback form

  • Use startOfSecond() for back-to-back records to eliminate millisecond differences

  • createFromFormat() uses current time for missing parts - chain startOfDay() for date-only

  • Never compare timestamps with assertEquals - use assertEqualsWithDelta or custom assertions

  • Seed random generators with fixed values - use DataProviders instead of random values

  • DataProviders run BEFORE Laravel boots - never use now(), config(), or facades in them

  • Never rely on database ordering without explicit ORDER BY

  • Shuffle test data deliberately to expose hidden dependencies

Mocking

  • Never use Mockery alias or overload mocks - use dependency injection instead

  • Use MockeryPHPUnitIntegration trait in base TestCase

  • Prefer Laravel fakes (Http::fake, Queue::fake) over manual mocks

  • Http::fake() precedence matters - specific routes first, wildcards last

  • Avoid Http::sequence() in parallel tests - responses are consumed unpredictably

  • Fake BEFORE factory creation to catch afterCreating hooks

  • Use partial fakes when possible - Bus::fake([SpecificJob::class]), Event::fake([SpecificEvent::class])

Cleanup & Isolation

  • Use DatabaseTransactions for test isolation

  • Custom cleanup BEFORE parent::tearDown() - Mockery and transactions close in parent

  • Clear caches BEFORE mocking, not after

  • Disable observable events in setUp for models with side effects

  • Use opt-in traits for targeted tearDown (RefreshesSushiModels, CleansUpTestFiles)

  • Prefer Storage::fake() over manual file cleanup

  • Reset global state (config, locale) in tearDown or at test end

Test Structure

  • Keep TestCase lean - only global concerns

  • Use traits for optional concerns with setUpXXX/tearDownXXX naming

  • Call Passport clients lazily, not in global setUp

  • Always specify scopes in Passport::actingAs()

  • Enable risky test detection in PHPUnit

Assertions

  • Use assertSame(true, ...) not assertTrue() to catch missing boolean model casts

  • Create custom assertions for value objects (Money, dates)

  • For Money: use Money->isEqualTo(), not assertEquals on floats

  • Use $this->fail() in try/catch blocks to prevent risky tests

  • Prefer expectException() over try/catch when only asserting exception type

  • Assert exception class AND message when relevant

Verification

  • Run suspected flaky tests many times: --repeat 100

  • Run in random order: --order-by=random

  • Run in parallel to expose race conditions

  • A test that fails once in 100 runs is a bug, not bad luck


The AI Agent Prompt

Why This Prompt?

AI coding assistants are increasingly writing and reviewing our tests. But they lack the hard-won knowledge of flaky test patterns - the alias mock trap, the DataProvider timing issue, the cache-before-mock ritual.

This prompt transfers that knowledge. Add it to your AI context and every test review catches the same issues a senior developer would.

Use it when:

  • AI writes tests for you - catch flakiness before commit

  • Debugging flaky failures - get pattern-based suggestions

  • Code review - automated checklist for test reliability

  • Onboarding - teach new devs through AI explanations

The Prompt

## Flaky Test Review Guidelines

When reviewing or debugging Laravel/PHP tests, check for these common flakiness patterns:

### Time-Related Issues
- [ ] Is time frozen with Carbon::setTestNow() or travelTo() at test start?
- [ ] Using travelBack() after travelTo(), or the callback form?
- [ ] Using startOfSecond() for back-to-back created records?
- [ ] createFromFormat() chains startOfDay() for date-only comparisons?
- [ ] DataProviders avoid now(), config(), facades? (They run before Laravel boots)

### Mock Issues
- [ ] Any Mockery alias:: or overload:: usage? (Use DI/Facades instead - these break parallel tests)
- [ ] Is MockeryPHPUnitIntegration trait used in base TestCase?
- [ ] Are fakes called BEFORE factory creation? (afterCreating hooks run immediately)
- [ ] Http::fake() has specific routes before wildcards?
- [ ] Avoiding Http::sequence() in parallel tests?
- [ ] Using partial fakes where possible? Bus::fake([Job::class]), Event::fake([Event::class])

### Isolation Issues
- [ ] Does test depend on database ordering without ORDER BY?
- [ ] Are caches cleared BEFORE mocking cached values?
- [ ] Are observable events disabled for models that trigger side effects?
- [ ] Is static state (singletons, static properties, Sushi connections) properly reset?
- [ ] Is global state (config, locale) reset in tearDown?

### tearDown Issues
- [ ] Custom cleanup runs BEFORE parent::tearDown()? (Mockery closes in parent)
- [ ] Are file handles closed before cleanup?
- [ ] Are opt-in cleanup traits used for specific concerns?

### Assertion Issues
- [ ] Using assertSame(true, ...) not assertTrue() for booleans? (Catches missing casts)
- [ ] Using Money->isEqualTo() not assertEquals for money values?
- [ ] Using $this->fail() in try/catch blocks? (Prevents risky tests)

### Common Fixes
1. Freeze time: `$this->travelTo(now()->startOfSecond())`
2. Disable events: `Model::ignoreObservableEvents(['created', 'updated'])`
3. Fake before factory: `Notification::fake(); $model = Model::factory()->create();`
4. Clear cache before mock: `cache()->forget('key'); Config::partialMock()`
5. Use Storage::fake() instead of manual file operations
6. Replace alias mocks with Factory Pattern or Facades

Copy this into your project as a TESTING_GUIDELINES.md or add it to your AI assistant context.


The complete series:

  1. The Reckoning - Why tests fail randomly

  2. Mock Madness - The alias mock trap

  3. The Determinism Principle - Time and randomness

  4. The Teardown Tango - Cleanup and assertions

  5. The Abstraction Avalanche - TestCase architecture

The End

300+ commits. 43 alias mocks eliminated. Countless flaky tests fixed.

And a TestCase that evolved from “throw everything in setUp to “make dependencies explicit.”

The test that passes locally and fails in CI? That's not a flaky test. That's a bug in your foundation.

Build it right. Keep it deterministic. Trust your tests.


The Flaky Test Chronicles ends here. Now go forth and make your tests boring. Boring tests are good tests.