Why Does the Same File Have a Different Hash?

You hash a file. A colleague hashes what they swear is the same file. The two digests don’t match. Or you hash a file today, hash it again after a round-trip through a build server, and the value has changed. Nothing was edited. So the first suspicion is the alarming one: is the hash function flaky?

It isn’t. A hash is one of the most reliable things in computing. The value changed because the bytes changed — quietly, in a way you didn’t see. This article is about all the ways “the same file” stops being the same bytes, and how to make them agree again.

The one idea: a hash is deterministic

A cryptographic hash is a pure function of the exact bytes you feed it. Same bytes in, same digest out — on every machine, in every language, on every run, forever. There is no randomness, no clock, no machine-specific seed. The SHA-256 of a given byte sequence is a constant of the universe.

Everything below assumes both sides ran the same algorithm and that you’re comparing the digests themselves — SHA-256 against SHA-256, not SHA-256 against MD5, and not hex against base64. Given that, the consequence is blunt. If two hashes differ, the two byte sequences differ. Full stop. There is no such thing as “the same file with a different hash.” What you have is two different byte sequences that you believe are the same file. The word “same” is doing the lying, not the hash.

So the question is never “why did the hash change.” It’s “what changed the bytes while I wasn’t looking.” The rest of this article is a tour of the usual culprits — most of them invisible when you open the file in an editor.

That second half is worth a moment, because it catches people. Make sure it’s the digest that differs, not just its printed form. AB… and ab… are one value in two letter cases, and the same digest can be shown in hex or in base64. If that’s your situation, the bytes are identical and nothing is wrong — that trap and its siblings live in why doesn’t my checksum match. From here on, assume the same algorithm and a genuine difference in the digest.

See it happen in thirty seconds

Before the taxonomy, watch the effect directly. Two lines of text, identical on screen, with Unix and then Windows line endings:

printf 'a\nb\n'     | sha256sum   # 4 bytes
# 911169ddaaf146aff539f58c26c489af3b892dff0fe283c1c264c65ae5aa59a2

printf 'a\r\nb\r\n' | sha256sum   # 6 bytes
# 58055bdcc73787eb88c78d36f0b4939e9c5dc1c3ad17e25cc85a6833cf1a0cab

Two extra bytes you cannot see, and the digest is unrecognizable. The same goes for a byte-order mark in front of the word hello:

printf 'hello'             | sha256sum
# 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824

printf '\xef\xbb\xbfhello' | sha256sum
# 7489ebbcc2a00056ddaaaac190bce473e5c03696ea1bd8ed83cf59a174283862

Run those yourself. Once you’ve seen three invisible bytes rewrite a digest, the rest of this article stops being surprising.

First: are you hashing the bytes you think you are?

Before blaming anything subtle, rule out the blunt version — you hashed something other than the file you meant.

• Wrong file, right name. A different build, a different version, a copy in another folder. Filenames lie; check the full path and the size.
• The file was still being written. You hashed it mid-download or mid-export, and a tool flushed more bytes a second later.
• A symlink or shortcut, not the target. You hashed the link itself, or a link that resolves to a different file on the other machine.
• A directory, not a file. A “hash of a folder” depends entirely on the tool: which files it includes, in what order, and whether it folds in metadata. Two tools give two answers for the “same” directory.

Confirm you’re pointing at the exact same object first. Often the size alone settles it: different sizes mean different bytes, and you can stop looking for anything clever.

Text: “the same text” is many byte sequences

This is where most of these mysteries live. Open two files in an editor and they look identical, character for character. But the hash reads bytes, not characters — and the same characters can be stored as very different bytes.

Character encoding. The letter A is one byte in UTF-8 and two in UTF-16. Accented and non-Latin text diverges much more. A file re-saved from UTF-8 to UTF-16, or to a legacy code page like Windows-1252 or Shift-JIS, is a different byte sequence with the same visible text. The file command will usually name the encoding on each side.

A byte-order mark (BOM). Some editors and exports — classic Windows Notepad, Excel’s CSV export — prepend a three-byte UTF-8 BOM (EF BB BF) to the front of the file. It’s invisible on screen. It changes every hash. One side has it, the other doesn’t, and the digests never line up.

Line endings. One of the most common causes across platforms. Unix ends a line with LF (\n); Windows uses CRLF (\r\n). Every line carries one extra byte on Windows, so a 100-line file differs by 100 bytes with not one visible change. Git makes this routine: with core.autocrlf on, it rewrites line endings on checkout and commit, so the working-tree file genuinely differs between a Windows clone and a Linux one.

A trailing newline. Editors and tools disagree about whether a file ends in a final newline. echo "x" appends one; printf x does not:

printf x     | sha256sum   # one byte
printf 'x\n' | sha256sum   # two bytes — a completely different digest

One extra \n at the end is one extra byte, and with a cryptographic hash like SHA-256 the avalanche effect makes the whole digest unrecognizable.

Text mode versus binary mode. A subtler one, and easy to miss because the file on disk never changes. Some hashing tools can read a file in “text mode” and translate line endings as they read, so the bytes that reach the hash aren’t the bytes on disk. GNU md5sum and sha256sum expose -b (binary) and -t (text); on Unix the two behave the same, but on Windows ports text mode is real and will silently normalize CRLF. The same trap exists in language APIs: opening a file in text mode instead of binary, then hashing what you read, hands the hash a translated and re-encoded stream. When hashing, always open in binary mode.

Invisible characters and normalization. Trailing spaces on a line. A tab swapped for spaces. A zero-width character or a non-breaking space pasted from a web page. And a subtle one — Unicode normalization. The character é can be a single code point (U+00E9) or an e followed by a combining accent (U+0301). They render identically and read as “the same” to a person, but they are different bytes.

One boundary worth drawing here: a normal file hash covers the file’s contents only — the filename is not part of it. So renaming a file never changes its content hash, and filename normalization doesn’t either. Normalization only bites when the name itself is the input: hashing a path string, a directory listing, a manifest of filenames, or archive metadata. It’s worth knowing because some macOS filesystems and tools normalize filenames while Linux generally preserves the raw bytes, so a path or filename list can differ across systems even when every file’s contents are identical.

Structured data: same meaning, different bytes

Text files aren’t the only thing that looks stable and isn’t. Hash a JSON document and you’re hashing one particular serialization of it, not the data. Anything that re-serializes — a different library, a different language, a round-trip through a database — can produce different bytes with identical meaning:

printf '{"a":1,"b":2}'    | sha256sum   # 43258cff783fe7036d8a43033f830adfc60ec037382473548ac742b888292777
printf '{"b":2,"a":1}'    | sha256sum   # 3fb75453225c732a76b7899ea2096dda1455189c89817239732182f73fe5a09f
printf '{"a": 1, "b": 2}' | sha256sum   # d8497d9d82770a70729261095aa98f7ef5154d7af499f8037b6ca250296785a6

Three digests, one piece of data. Key order changed in the second; only whitespace changed in the third. Other serializers vary further — escaping non-ASCII as \uXXXX or emitting it raw, trailing newlines, how floats are printed, whether the encoder sorts keys at all.

The fix isn’t to hash harder. It’s to canonicalize before hashing: agree on one serialization and produce it on both sides. Sort keys, drop insignificant whitespace, pin the number and escaping rules. JSON has a standard for exactly this — JSON Canonicalization Scheme, RFC 8785 — and most ecosystems have a library for it. Hash the canonical form, and two systems that agree on the data will agree on the digest.

Archives: same contents, different container

Zip and tarball files add their own layer. Two archives can hold byte-for-byte identical files and still hash differently, because the container records more than the files:

• Timestamps. Most formats store each entry’s modification time. Repack a minute later and the archive bytes change.
• File order. The order entries are added is part of the archive. Two runs that walk the directory differently produce different bytes.
• Compression settings. A different level, or a different zlib/gzip version, yields different compressed bytes for identical input.
• Metadata. Permissions, owner and group IDs, and — for gzip — the original filename and modification time baked into the header.

This catches more people than it should, because many formats you don’t think of as archives are ZIP containers: .docx, .xlsx, .pptx, .jar, .epub. Open a document, change nothing, save it again, and the file hashes differently — new timestamps and possibly a new entry order inside the zip, even though every part is unchanged.

This is one important reason reproducible builds are hard — toolchain versions, environment, and build paths matter too — and it’s why tools grew flags to strip the noise: gzip -n drops the name and time, and GNU tar can pin order and time with tar --sort=name --mtime='UTC 1970-01-01' … (those are GNU options; the BSD tar shipped with macOS doesn’t support all of them). If you’re hashing an archive to compare builds, hash the extracted contents instead — there’s a recipe for that below.

A quick reference

How to actually find the difference

Stop re-hashing and start comparing bytes.

The commands below assume a Linux or macOS shell, and a few differ by platform. sha256sum is GNU — on macOS use shasum -a 256. cat -A is GNU-only; BSD cat on macOS offers -e and -t instead. And diff <(…) needs bash or zsh process substitution. On Windows, fc /b compares two files byte by byte, and PowerShell’s Format-Hex dumps bytes.

1. Compare sizes first. ls -l, or wc -c. Note what a size difference does and doesn’t tell you: it gives the net change in length, not the number of changed bytes — swap 1,000 bytes for 1,000 others and the size is identical. Still, the delta is a useful hint: a few bytes suggests a BOM or a trailing newline; a gap that scales with the line count points at CRLF.
2. Look at the bytes. xxd yourfile | head, or diff two dumps with diff <(xxd a) <(xxd b). The first differing offset narrows down where the byte-level difference begins — which is a starting point, not automatically the root cause. An encoding change alters everything after it, CRLF re-offends on every line, and an archive’s first difference is often just a timestamp.
3. Reveal the invisible. cat -A shows line endings and trailing spaces, and a peek at the first three bytes catches a BOM. file will guess the encoding, though for UTF-8 without a BOM and some legacy code pages it is a heuristic, not a verdict.

If you need to recompute a digest along the way — to check one side after a change, or to try a different algorithm — drop the file into the hash generator; it computes one algorithm at a time, in hex. But the difference is found with sizes and a hex dump, not by hashing again.

Make it stop: normalizing so the hashes agree

Finding the byte is half the job. If two sides keep drifting, normalize them so the digests match by construction.

One note before the commands: sha256sum is GNU and is not on a stock macOS, where the equivalent is shasum -a 256. That substitution applies to every snippet below.

Line endings. Convert CRLF to LF before hashing. dos2unix does it if installed — it isn’t there by default on macOS — and tr is POSIX, so it works anywhere:

tr -d '\r' < winfile.txt | sha256sum       # Linux
tr -d '\r' < winfile.txt | shasum -a 256   # macOS

On the file from earlier, that returns 911169dd…: exactly the digest of the LF version. (tr -d '\r' drops every carriage return, not only the ones ending a line — for well-formed text files those are the same set.) To stop the drift at its source in a Git repo, commit a .gitattributes with * text=auto eol=lf so everyone’s working tree agrees, instead of relying on each developer’s core.autocrlf.

A UTF-8 BOM. A UTF-8 BOM is exactly three bytes at the front, so skipping those three is the simplest fix — no regex escapes involved. Other encodings differ: a UTF-16 BOM is two bytes, a UTF-32 BOM is four. The commands below assume you have already confirmed a UTF-8 BOM is present — run them on a file that doesn’t have one and you’ll chop off three real bytes:

tail -c +4 utf8-bomfile.txt | sha256sum         # Linux
tail -c +4 utf8-bomfile.txt | shasum -a 256     # macOS

That returns 2cf24dba… — the digest of plain hello, BOM removed.

Encoding. Convert both sides to one encoding first:

iconv -f UTF-16 -t UTF-8 input.txt | sha256sum

Directories and archives. Don’t hash the container. Hash the contents, then hash the sorted list of those hashes — that drops timestamps, packing order, and compression settings in one step. Run it from inside the directory so the recorded paths are relative:

# Non-deterministic: the container carries timestamps and entry order
tar -cf - mydir | sha256sum

# Deterministic: contents plus relative paths, order pinned by sort
( cd mydir && find . -type f -exec sha256sum {} + | sort | sha256sum )

Be precise about what that digest covers, because sha256sum prints the path next to each hash and the path is therefore part of the input. It hashes file contents plus their relative paths. It ignores directory timestamps and packing order — but it changes if a file is renamed or moved, it skips empty directories, and it captures nothing about symlinks, permissions, or ownership. Run it from outside (find mydir …) and the folder’s own name lands in the hash too, so two identical trees under different folder names won’t match. For “did these two trees end up with the same files, in the same places?” this is exactly right. If you also need permissions and symlinks, reach for a tool built for that.

The checklist

1. Same object? Same path, same size, not a directory, and not a file still being written.
2. A real digest difference? Or just upper- vs lower-case hex, or hex vs base64 — which is no difference at all, and is covered in the checksum-mismatch guide linked above.
3. Hashing text? Suspect encoding, a BOM, LF vs CRLF, a trailing newline, invisible whitespace, or Unicode normalization — and make sure your tool or your code is reading in binary mode.
4. Hashing structured data? JSON and friends have many valid serializations. Canonicalize first, then hash.
5. Hashing an archive? Suspect timestamps, file order, compression, and metadata — including in .docx, .xlsx, and .jar, which are zips. Hash the contents, not the container.
6. Still stuck? Compare sizes, then hex-dump and diff. The first differing byte helps narrow down where to investigate; it does not automatically reveal the root cause.

The through-line is the fact from the top: a hash is deterministic, so a different hash is proof the bytes differ. “Same file, different hash” is never a hash problem. It’s a file that quietly became a different sequence of bytes — a newline, an encoding, a BOM, a re-serialization, a repack — and the hash, doing exactly its job, refused to pretend otherwise. Find the byte that changed, then normalize so it can’t change again.