Please ensure Javascript is enabled for purposes of website accessibility

GPT-4 Answers Tax Questions, Gets Them Mostly Right

screenshot of Ask an AI Accountant answering a tax question about gambling winnings

These days there is a lot of yapping about what AI can and can’t do; it CAN answer bar exam questions so well it’s nearly in the 90th percentile of test-takers, it CAN’T search in real time to find up-to-date answers beyond what it’s been trained on so it won’t be getting stuck in a late-night Wikipedia hole any time soon. Today we’ve learned there’s one more can: it CAN answer tax questions and be broadly correct 84.19% of the time.

The folks at tax app Keeper trained GPT-4 on 2023 tax updates and then set the public loose on it, inviting them to ask their burning tax questions. From there, actual human professionals fact-checked the answers. For example:

screenshot of Ask an AI Accountant tool

One of the reviewers was Isaiah McCoy, a CPA working in Miami. Going into it he tempered his expectations and thought the tool might hit 60/40 right/wrong or even 50/50 just because tax law is so nuanced. “It far exceeded my expectations,” he said. “Its success rate was more like 80/20 or 90/10. I think it did a great job overall, really blew me away.” As for the prospect of getting replaced by AI, he says he feels moderately safe. “I definitely feel like it’s a threat or an opportunity depending on how you look at it,” he said.

By evaluating 215 answers, Keeper learned that the AI does a good job of:

  • Knowing which tax forms apply to which situation
  • Understanding real estate tax implications
  • Naming deductible business expenses for specific industries

And it sucks at:

  • Getting tax form details right, including what certain lines and boxes are intended for
  • Answering questions about tax year applicability
  • Addressing state and local tax nuances

Here’s one it bombed at:

Screenshot of Ask an AI Accountant tax tool

It’s wild how it matched the casual tone of the original question in its response with a “Hey there!”

Feel free to give it a spin yourself, just remember not to input any PII. And don’t threaten it, you don’t want to be on its shit list when the inevitable takeover happens.

Latest Accounting Jobs--Apply Now:

Have something to add to this story? Give us a shout by email, Twitter, or text/call the tipline at 202-505-8885. As always, all tips are anonymous.

Related articles

PwC Australia is Very Very Sorry, You Guys

Presumably because the many apologies and decisions made before this letter have not sufficiently gotten the heat off their backs (and boy is it hot), PwC Australia published an open letter apology on their website Monday. The entire text, including the formatting, appears in below. At issue, if you make your residence under a rock, […]

Collie carrying American flag in its mouth

Friday Footnotes: The ERC Problem; EY Double Dips; Stress Less in Public Accounting | 5.26.23

Footnotes is a collection of stories from around the accounting profession curated by actual humans and published every Friday at 5pm Eastern. While you’re here, subscribe to our newsletter to get the week’s top stories in your inbox every Tuesday and Friday. Have a safe Memorial Day. Long Read How a Pandemic-Era Program Became a […]