These days there is a lot of yapping about what AI can and can’t do; it CAN answer bar exam questions so well it’s nearly in the 90th percentile of test-takers, it CAN’T search in real time to find up-to-date answers beyond what it’s been trained on so it won’t be getting stuck in a late-night Wikipedia hole any time soon. Today we’ve learned there’s one more can: it CAN answer tax questions and be broadly correct 84.19% of the time.
The folks at tax app Keeper trained GPT-4 on 2023 tax updates and then set the public loose on it, inviting them to ask their burning tax questions. From there, actual human professionals fact-checked the answers. For example:
One of the reviewers was Isaiah McCoy, a CPA working in Miami. Going into it he tempered his expectations and thought the tool might hit 60/40 right/wrong or even 50/50 just because tax law is so nuanced. “It far exceeded my expectations,” he said. “Its success rate was more like 80/20 or 90/10. I think it did a great job overall, really blew me away.” As for the prospect of getting replaced by AI, he says he feels moderately safe. “I definitely feel like it’s a threat or an opportunity depending on how you look at it,” he said.
By evaluating 215 answers, Keeper learned that the AI does a good job of:
- Knowing which tax forms apply to which situation
- Understanding real estate tax implications
- Naming deductible business expenses for specific industries
And it sucks at:
- Getting tax form details right, including what certain lines and boxes are intended for
- Answering questions about tax year applicability
- Addressing state and local tax nuances
Here’s one it bombed at:
It’s wild how it matched the casual tone of the original question in its response with a “Hey there!”
Feel free to give it a spin yourself, just remember not to input any PII. And don’t threaten it, you don’t want to be on its shit list when the inevitable takeover happens.
[I]f we are going to make real progress, we can’t fixate on every overhyped, half-baked tax slogan that comes along. Sooner or later we must get back to basics. Here’s the main question: Should taxes be cut, raised, or reformed without changing overall revenue? The answer is that taxes should be cut in the short term, raised after we are clearly out of our cyclical downturn, and then reformed only after we have settled on the magnitude of tax increases needed for deficit reduction. [
They aren’t exactly the U.S. Treasury and don’t foresee any populist outrage but Miami Heat Limited Partnership did Tim a fave and bought his 7,500 square foot manse for $1.985 million, according to