There is no hard and fast rule, but the best approach is to reduce the apparent complexity for the user, using the best mapping of down and up arrows to the terminology "Ascending" and "Descending".
Note that most non-numeric concepts doesn t have a strong natural mapping to "up" or "down".
Do letters/words go "up" or "down" in the dictionary? How about dates and times? Where it is ambiguous, I believe there is no "right" answer - I recommend allowing "Descending" to represent the more useful sort order for consistency, as the user is likely to think about the table as being "down" as they move their eyes down. Leaving aside digital representations since the users of most applications do not know or care about the internal representation. Therefore alphabet sorting could be "Descending" for A-Z, and time could be Descending with most recent first. The good news is, as long as the first click gives the user the sorting behavior that they expect, and the second reverses it, they mostly won t care which of the two modes is used.
That challenge about the correct sort to do "first" (on showing the content, or sorting on a new column) is the more important implication of the question. The type AND intended usage of the information determines it should sort. Alphabetic should always default A-Z, "descending" by my above logic. Numbers vary much more by use: numbers that represent sequential identifiers, say, an employee ID would be ascending (1-10), while sorting on quantities or price would usually be descending, to bring the largest values to the top. Time also varies - most recent first ("Descending") usually works but in some contexts, the oldest should be listed first.