AI reverse engineering: test REA beyond the demo

Why it matters

A convincing app clone can hide a substantial maintenance problem. The interface may look familiar while imports, error handling, saved state, and unusual inputs behave differently. If you build integrations or maintain older software, those differences are where the actual engineering work begins.

My strongest takeaway from this episode of The Next New Thing is the value of giving agents better ways to inspect and constrain their work. REA supports investigation, visual tools provide editable structures, and developer skills supply more focused guidance. Each deserves a place in a workflow when it improves an outcome you can check.

I would approach the promise of copying an app as a testable claim. Below is an evaluation plan that adds something beyond the demonstrations: a controlled comparison, explicit acceptance cases, and a change request after the first successful run. This is a proposed experiment, not a claim that Luiz has already benchmarked these repositories.

Key points

• 03:53 A working clone needs a definition of behavioral equivalence. The 2048 demonstration establishes that a visible interaction can be reproduced. A work application needs broader checks: invalid input, state transitions, persistence, and rules that screenshots cannot reveal. Those checks determine how much of the application was actually understood.
• 15:01 The Ghidra demo exposes a secret embedded in a test program. The presenters create the target themselves and recover a code stored inside it. That illustrates a specific implementation weakness rather than a universal password recovery capability. Decompilation is among the capabilities documented by the official Ghidra project.
• 04:59 Specialized tooling has to earn its setup cost. The episode mentions a Preview reproduction where adding REA did not improve the result. That observation deserves attention alongside the successful demonstrations. A useful evaluation needs a baseline showing what the existing agent can already accomplish.
• 07:46 Passing tests are meaningful only when the expected behavior is sound. The TDD discussion highlights the importance of seeing the failure before implementation. The SwiftUI skill discussed at 23:20 addresses a related concern: supplying relevant technical guidance. Both depend on the quality of the reference material and the judgment used to review it.
• 13:55 Visual constraints can reduce revision work. Diagram Design starts from simpler structures that can be customized. HyperFrames, demonstrated at 11:41, brings code-based control to video creation. I would evaluate both through a precise revision request after the initial output.
• 18:22 A semantic summary may discard the exception you need later. The Billion Context discussion concerns summarizing a growing history, which can lose detail. It should not be read as a claim that all compression is lossy. During reverse engineering, a missing exception can become an incorrect implementation rule.
• 21:43 Physical prototypes make verification unavoidable. The Text to CAD discussion emphasizes measuring, printing, and adjusting the result. Software needs an equally concrete feedback loop: representative inputs, observable outputs, and deliberate attempts to expose incorrect assumptions.
• 24:58 Ownership includes responsibility for keeping the application useful. The openGym discussion centers on software serving the user's tracking goals. AnyPS5, discussed at 05:32, offers a different perspective through an ambitious compatibility effort with substantial limitations in the episode. Available code needs to be assessed alongside compatibility, upkeep, and daily usability.

How to apply

AI reverse engineering: run a one-hour evaluation

Start with a small application you own and understand. A utility that imports CSV files and calculates totals is a useful target. Keep its source as a review reference outside the material supplied to the agent. Give the agent the runnable application, synthetic input files, and a narrow description of the behavior to investigate.

REA connects agents to inspection tools for applications, native binaries, and runtime behavior. Its documentation distinguishes JavaScript inspection from native analysis. Pick the route that matches your target before spending time on dependencies.

For an initial evaluation, allocate ten minutes to defining the behavior, fifteen to your usual agent, twenty to a fresh session using REA, and fifteen to reviewing both results. This timebox should reveal whether a longer investigation is justified. It is a proposed starting budget, not an estimate of how long arbitrary applications take to reconstruct.

Build an answer key the agent cannot invent

Choose cases that reveal actual business rules. For a CSV importer, a compact test set might look like this:

| Input | What to verify | | --- | --- | | Accented customer name | Text survives import correctly | | Locale-specific number | The value follows the application's defined convention | | Empty file | The expected empty-input behavior appears | | Duplicate row | The original duplicate-handling rule is preserved | | Invalid amount | The error is reported without damaging valid records |

Establish expected results by running the original application and checking its source. An agent that invents a rule and then tests its implementation against that rule has not established equivalence.

For teams handling Brazilian data, include decimal commas, thousands separators, and accented text. For other environments, use the formats your users actually encounter. The important property is that the fixtures expose assumptions which a polished interface would conceal.

Ask for an investigation record

I would begin with a request like this:

Investigate only this application's CSV import behavior. Record observed inputs, outputs, and error messages. Separate observations from hypotheses. Attach available evidence to each conclusion and list anything you could not verify. Then propose an independent implementation and tests for the confirmed behavior.

That record helps locate mistakes. You can distinguish a misunderstood rule from a coding error instead of treating every failure as a reason for another broad prompt.

When inspecting a binary, treat decompiled output as analysis material. Original names, comments, and project structure may be unavailable. If the investigation reveals an embedded secret, record that finding separately from functional reproduction. It answers a different question about the application.

Measure the review burden

Use the same model, fixtures, and acceptance criteria for both attempts. Record tool versions, setup time, execution time, model cost where available, and the number of human interventions. Run separate sessions so the second attempt does not inherit discoveries from the first.

Review failed cases as carefully as successful ones. An importer that handles normal rows but silently drops malformed records can create more work than it saves. Ask the agent to show the evidence behind each implemented rule, then inspect a sample yourself.

If you use TDD for the reconstruction, proceed one behavior at a time: write a test, observe the expected failure, implement enough to pass, and refactor while preserving the result. Add human-reviewed acceptance cases to check whether the problem itself was understood correctly.

Make one change before committing

After the first working version, request an additional CSV column or a more specific validation rule. Check whether previous cases still pass and how much context must be supplied again. This exposes the maintenance quality of the reconstructed application.

The same evaluation question applies to the episode's other tools. Can a rendered video change one displayed price cleanly? Does a diagram remain readable after adding a step? Does a CAD model preserve dimensions after rounding its corners? Can a summarized investigation still point back to the original evidence?

I would adopt REA after seeing a repeatable improvement on a relevant task: more correct behavior, less manual investigation, or faster review. Keep the fixtures, evidence, and known limitations with the deliverable. They give the next person a practical way to verify future changes.