Compare Two Text Files in Python Without Modifying Them

Batu Lab Notes · Batu

When comparing two versions of a configuration or notes file, “are they the same?” is only the first question. The bytes may be identical, the visible text may be equal despite different line endings, or the content may contain a real change. For small UTF-8 text files, Python’s standard library can show these distinctions without writing to either input.

Quick answer

For small, stable UTF-8 files, read bounded bytes, compare them first, then decode and produce a bounded unified diff. Decide explicitly whether newline differences matter. Keep both inputs read-only and review the diff for private data before sharing it.

Set a limit before reading

Loading an input of unknown size directly into memory is a poor default. This example checks both sizes with Path.stat(), then performs a bounded read and applies a 256 KiB limit per file. It rejects directories and other non-regular paths. If reading raises OSError, the result includes the exception type but does not copy a potentially sensitive local path from the exception message.

This is not a security guarantee. A path can change between the metadata check and the open operation. The bounded read still prevents that operation from consuming more than max_bytes + 1, but hostile or concurrently replaced files require stronger operating-system controls. This example is intended for small, stable files.

Byte equality and text equality answer different questions

A bounded Path.open("rb") read returns raw bytes. Equal byte sequences mean the files are byte-for-byte identical. For text comparison, the example decodes those bytes as strict UTF-8. It does not guess another encoding after a failure; it returns not_utf8. Python also provides difflib.diff_bytes() for byte sequences with unknown or inconsistent encodings, but such output remains raw content and still requires careful interpretation.

Line endings demonstrate the distinction. Windows-style CRLF (\r\n) and LF (\n) are different bytes. The code normalizes CRLF and standalone CR to LF for its text view. If line endings are the only difference, it reports byte_equal=False and newline_only=True. Remove this normalization when exact line-ending style is meaningful to your application.

Keep the unified diff bounded

The official difflib documentation describes unified_diff() as a compact before-and-after representation with surrounding context. The example uses two context lines, labels the inputs before and after instead of exposing paths, and limits the returned diff to 20,000 characters. diff_truncated=True makes any cut explicit.

A file whose final line has no newline needs special handling. The helper adds a visible ⟪no newline at end of file⟫ marker to the comparison view, so the last changed line does not run into the next diff control line.

Review a raw diff before sharing it. Changed lines can contain names, email addresses, access tokens, customer data, or other private information. This function does not redact them. Keeping paths out of the labels reduces one source of leakage; it does not sanitize file contents.

Copy the complete checker

Save the following complete block as batu_file_compare.py:

"""Bounded, read-only comparison for two small UTF-8 text files."""
from __future__ import annotations

import difflib
from pathlib import Path

MAX_BYTES = 256 * 1024
MAX_DIFF_CHARS = 20_000


def compare_text_files(left, right, *, max_bytes=MAX_BYTES):
    paths = (Path(left), Path(right))
    if not isinstance(max_bytes, int) or isinstance(max_bytes, bool) or max_bytes < 1:
        raise ValueError("max_bytes must be a positive integer")

    try:
        sizes = [path.stat().st_size for path in paths]
        if any(not path.is_file() for path in paths):
            return {"status": "unreadable", "error": "regular_file_required"}
        if any(size > max_bytes for size in sizes):
            return {"status": "too_large", "sizes": sizes, "limit": max_bytes}
        raw_files = []
        for path in paths:
            with path.open("rb") as stream:
                raw = stream.read(max_bytes + 1)
            if len(raw) > max_bytes:
                return {"status": "too_large", "sizes": sizes, "limit": max_bytes,
                        "grew_during_read": True}
            raw_files.append(raw)
        raw_left, raw_right = raw_files
    except OSError as error:
        return {"status": "unreadable", "error": type(error).__name__}

    byte_equal = raw_left == raw_right
    try:
        text_left = raw_left.decode("utf-8")
        text_right = raw_right.decode("utf-8")
    except UnicodeDecodeError:
        return {"status": "not_utf8", "byte_equal": byte_equal}

    normalized_left = text_left.replace("\r\n", "\n").replace("\r", "\n")
    normalized_right = text_right.replace("\r\n", "\n").replace("\r", "\n")
    text_equal = normalized_left == normalized_right
    def diff_lines(text):
        lines = text.splitlines(keepends=True)
        if lines and not lines[-1].endswith("\n"):
            lines[-1] += "\n"
            lines.append("⟪no newline at end of file⟫\n")
        return lines

    delta = "".join(difflib.unified_diff(
        diff_lines(normalized_left), diff_lines(normalized_right),
        fromfile="before", tofile="after", n=2,
    ))
    truncated = len(delta) > MAX_DIFF_CHARS
    return {
        "status": "equal" if text_equal else "different",
        "byte_equal": byte_equal,
        "newline_only": text_equal and not byte_equal,
        "diff": delta[:MAX_DIFF_CHARS],
        "diff_truncated": truncated,
    }

In the same directory, create before.txt containing status: draft and after.txt containing status: reviewed. End both files with a newline. Then save or run this caller:

from batu_file_compare import compare_text_files

result = compare_text_files("before.txt", "after.txt")
if result["status"] == "different":
    print(result["diff"])
else:
    print(result)

The checker opens both inputs only in binary read mode. It does not rename, convert, truncate, or save them. The files used to set up the example are separate inputs; comparison itself remains read-only.

Interpret the result narrowly

equal means the text is equal under the selected UTF-8 decoding and newline normalization. It does not prove that the files have the same origin, are safe, or contain correct configuration. different means a textual difference exists; it does not decide which version is correct.

A practical sequence is: check type and size, compare bytes, decode UTF-8, then inspect the bounded diff. A human still needs to decide whether a change is expected and whether the output is safe to share.

Related guide

Comparing exports is only one part of validation. For structured data, also run these seven CSV quality checks before importing data; a readable diff does not prove a valid schema or unique IDs.

Official sources

Disclosure: This article was prepared with AI assistance. Its technical claims were checked against the official Python documentation, and the example was verified with synthetic local data.

Comments

Post a Comment

Popular posts from this blog

Seven CSV Quality Checks to Run Before Importing Data

Batu Lab Notes · Batu A CSV file is not ready for import merely because it opens. The wrong delimiter can collapse an entire row into one column. An unexpected encoding can damage non-ASCII characters. A repeated identifier can overwrite an existing record in the destination. A short, read-only check makes these problems visible before data reaches a spreadsheet or application. Quick answer Before importing a CSV, set its encoding and delimiter, validate the exact header and every row width, distinguish empty IDs, duplicate IDs, and duplicate rows, and keep the check read-only. 1. Specify the encoding Python may open a text file with the platform’s default encoding, which can differ across machines. If the expected encoding is UTF-8, encoding="utf-8-sig" is a practical choice: it reads ordinary UTF-8 and skips an optional UTF-8 byte order mark at the start. If decoding fails, verify the source application’s export settings instead of silently trying unrelat...
Read more

Explain why SQLite transactions cannot make an HTTP call atomic

Direct answer A SQLite transaction can atomically commit database statements handled by that connection. It cannot include an HTTP service in the same commit protocol. The example commits an outbox row first, then simulates a network failure. The durable row remains pending , which is exactly the information a later delivery worker needs instead of pretending both actions succeeded or failed together. The edge case is failure after the database commit but before remote acknowledgement. Retrying then requires an idempotency design owned by the remote endpoint or a stable request key; it is not supplied by SQLite’s COMMIT . Conversely, sending the request before storing a durable intent can leave no local evidence when the process stops. This demonstration intentionally does not make an HTTP request, implement an outbox dispatcher, or prove delivery exactly once. It shows the boundary that must be designed around. Use transactions to protect the local state transition, and test remote...
Read more