Question

It s difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.

Closed 11 years ago.

The following python texts the first occurrence of a line in a file.

#!/usr/bin/env python
import sys

x = set() 

for line in sys.stdin:
    if line not in x:
        print line,
        x.add(line)

uniq only works for adjacent lines. The file this will be used on is very large so sort | uniq is not ideal. Is there a standard tool that does this?

Answer 1

您是否尝试了<代码>-u? 至少,uniq man page 。

Answer 2

using `awk`

awk  !x[$0]++  file.old > file.new

using awk

友情链接

using `awk`