With a real high action, as you get further from the nut some fretted notes can sound a bit out of tune compared to the open string, because you are stretching/bending the string more when pushing it all the way down to the fretboard. In contrast, when the string is already very close to the fretboard as with a low action, fretting it does not stretch or bend the string much. Some folks don't notice the intonation difference though, it's usually subtle.
Another possible downside is the 'trampoline' feel of high action paired with nylon strings....not unlike walking on a trampoline. ;)
I would point out that what Strum mentions may only be an issue with flush frets. Your intonation is a result of the correct pressure, the angles your finger presses the string with, and sensing that correct spot. Height of the string is of no consequence.