English 中文(简体)
如何使用哈希来比较两个目录中的许多文件,并找出哪些是相同的?
原标题:How to use hashing to compare many files from two directories and find which are identical?

我有一个数据集,我正在从在线驱动器迁移到现场驱动器。我也在做数据管理,所以我一边重组和重命名文件,以符合我们的新标准(此时不编辑任何数据)。这是所有的GIS数据,在人们在这台驱动器和工作电脑上的各种项目中都有参考。移动或更新这些文件最终会破坏基本上所有文件的源代码,所以我需要找到一种方法来替换它们。

我已经想好了如何替换源文件路径,但这需要我知道新旧文件路径是什么。我唯一能想到的另一种方法是手工复制旧的和新的,或者在我走的时候把它们匹配起来,这……并不理想。我看过很多关于比较单个文件的东西,但我一辈子都找不到关于比较两个文件夹的文件或类似性质的东西。这个问题可能在其他地方得到了答案,但我找不到,所以如果你已经知道这里的答案,请给我一个答案。

附加信息:我们拥有的数据总量约为150 gb,但可能有1/3不需要处理,可能会被排除在外,因为它的内容包括图像和积压的东西,这些东西很可能永远不会被再次使用和删除。

提前感谢!

问题回答

只需使用rsync。它为您在两个目录之间进行所有这些文件同步。

rsync通常用于远程复制,但也可以用于本地复制。

或者,如果你需要经常这样做,只需将所有文件检查到Git中,然后复制到另一台机器或另一个目录只需Git pull/push即可。你可以在没有Git服务器的情况下做到这一点,因为Git是一个分布式VCS。Git可以检测重命名和内容重复的文件,所以这种操作会很快。





相关问题
Can Django models use MySQL functions?

Is there a way to force Django models to pass a field to a MySQL function every time the model data is read or loaded? To clarify what I mean in SQL, I want the Django model to produce something like ...

An enterprise scheduler for python (like quartz)

I am looking for an enterprise tasks scheduler for python, like quartz is for Java. Requirements: Persistent: if the process restarts or the machine restarts, then all the jobs must stay there and ...

How to remove unique, then duplicate dictionaries in a list?

Given the following list that contains some duplicate and some unique dictionaries, what is the best method to remove unique dictionaries first, then reduce the duplicate dictionaries to single ...

What is suggested seed value to use with random.seed()?

Simple enough question: I m using python random module to generate random integers. I want to know what is the suggested value to use with the random.seed() function? Currently I am letting this ...

How can I make the PyDev editor selectively ignore errors?

I m using PyDev under Eclipse to write some Jython code. I ve got numerous instances where I need to do something like this: import com.work.project.component.client.Interface.ISubInterface as ...

How do I profile `paster serve` s startup time?

Python s paster serve app.ini is taking longer than I would like to be ready for the first request. I know how to profile requests with middleware, but how do I profile the initialization time? I ...

Pragmatically adding give-aways/freebies to an online store

Our business currently has an online store and recently we ve been offering free specials to our customers. Right now, we simply display the special and give the buyer a notice stating we will add the ...

Converting Dictionary to List? [duplicate]

I m trying to convert a Python dictionary into a Python list, in order to perform some calculations. #My dictionary dict = {} dict[ Capital ]="London" dict[ Food ]="Fish&Chips" dict[ 2012 ]="...

热门标签