Please ensure Javascript is enabled for purposes of website accessibility

GPT-4 Answers Tax Questions, Gets Them Mostly Right

screenshot of Ask an AI Accountant answering a tax question about gambling winnings

These days there is a lot of yapping about what AI can and can’t do; it CAN answer bar exam questions so well it’s nearly in the 90th percentile of test-takers, it CAN’T search in real time to find up-to-date answers beyond what it’s been trained on so it won’t be getting stuck in a late-night Wikipedia hole any time soon. Today we’ve learned there’s one more can: it CAN answer tax questions and be broadly correct 84.19% of the time.

The folks at tax app Keeper trained GPT-4 on 2023 tax updates and then set the public loose on it, inviting them to ask their burning tax questions. From there, actual human professionals fact-checked the answers. For example:

screenshot of Ask an AI Accountant tool

One of the reviewers was Isaiah McCoy, a CPA working in Miami. Going into it he tempered his expectations and thought the tool might hit 60/40 right/wrong or even 50/50 just because tax law is so nuanced. “It far exceeded my expectations,” he said. “Its success rate was more like 80/20 or 90/10. I think it did a great job overall, really blew me away.” As for the prospect of getting replaced by AI, he says he feels moderately safe. “I definitely feel like it’s a threat or an opportunity depending on how you look at it,” he said.

By evaluating 215 answers, Keeper learned that the AI does a good job of:

  • Knowing which tax forms apply to which situation
  • Understanding real estate tax implications
  • Naming deductible business expenses for specific industries

And it sucks at:

  • Getting tax form details right, including what certain lines and boxes are intended for
  • Answering questions about tax year applicability
  • Addressing state and local tax nuances

Here’s one it bombed at:

Screenshot of Ask an AI Accountant tax tool

It’s wild how it matched the casual tone of the original question in its response with a “Hey there!”

Feel free to give it a spin yourself, just remember not to input any PII. And don’t threaten it, you don’t want to be on its shit list when the inevitable takeover happens.