Net question and answer: is there the most efficient way to check large files in C?

Consultation area

Dario：

I need to synchronize large files between multiple machines , But the file is as high as 6G, I usually synchronize manually every few weeks , Considering that the file name of the file often changes , To check consistency , I'm thinking about using  checksum  Mechanism .

My plan is to   Source machine   and   Target machine   Do on   The checksum  , Then in the machine room copy Bring a checksum when you file , So as to judge the integrity of the document , I try to do this with the following code .

        static string GetChecksum(string file)
        {
            using (FileStream stream = File.OpenRead(file))
            {
                SHA256Managed sha = new SHA256Managed();

                byte[] checksum = sha.ComputeHash(stream);

                return BitConverter.ToString(checksum).Replace("-", String.Empty);
            }
        }

Now the problem is ：

• SHA256 In mode ,1.6G Documents need to be consumed 20 Minutes is the checksum .

• MD5 In mode ,1.6G Documentation needs 6.15 Minutes is the checksum .

Is there a more efficient way to calculate The checksum Well ？

Answer area

Anton Gogolev：

The problem is  SHA256Managed  Only... Can be read at a time 4096 byte, This is for disk IO Throughput is too small .

To speed up , It can be used BufferedStream To the parcel FileStream, So as to improve FileStream default 4096 Size , However, this value can set a reasonable range according to your own scenario , Here I set it to  1M.

// Not sure if BufferedStream should be wrapped in using block
using(var stream = new BufferedStream(File.OpenRead(filePath), 1024 * 1024))
{
    // The rest remains the same
}

It's on my machine .

• SHA256 In mode ,2G Documentation needs 2 minute Calculate the check sum .

• MD5 In mode ,2G Documentation needs 1 minute Calculate the check sum .

Fabske：

You can get to know  XxHash.Net, github Address ：https://github.com/wilhelmliao/xxHash.NET

and  xxHash  The algorithm seems to be the fastest , Here is xxHash Of benchmark chart .

Specific reference github：https://github.com/Cyan4973/xxHash

Comment area

Speaking of  checksum, It reminds me of Principles of computer network , Feeling is the most important word in this class ,TCP,UDP Bao wubu checksum, Its function is mostly afraid of losing some bytes due to interference in network transmission , such checksum Can accurately find , But I remember redis Of xxx.rdb There are also  checksum  Of , This thing is really too important .