Bytes Before Meaning

Reading a file looked like one line of code: open it, read it, done. Working through file handling and data tools in Python, I kept running into the same surprise. "Read the file" is really several steps, and each step makes a decision about what the data means. Bytes become numbers or text. Text becomes lines. Lines become rows and columns. If one of those decisions is wrong, nothing crashes. The answer is just quietly different.

That matters to me because in digital forensics the file is the evidence. A tally, a hash match or a chart is only as good as the reading underneath it. So this piece follows one small made-up example from raw bytes up to a saved chart, and notes where each layer could mislead me.

Layer one: bytes have no meaning yet

Here is a tiny invented container format: four bytes of "magic" that identify it, a 2-byte version number and a 4-byte record count. I wrote it with struct and read it back in binary mode:

import struct

# Build a tiny made-up container: 4-byte magic, 2-byte version, 4-byte record count.
header = b"LOG1" + struct.pack("<HI", 2, 300)
with open("sample.bin", "wb") as f:
    f.write(header)

with open("sample.bin", "rb") as f:
    raw = f.read(10)

print(raw)
print(raw[:4] == b"LOG1")
print(struct.unpack("<HI", raw[4:]))
print(struct.unpack(">HI", raw[4:]))

b'LOG1\x02\x00,\x01\x00\x00'
True
(2, 300)
(512, 738263040)

Three things stood out.

First, the same six bytes give two completely different answers. < means little-endian (least significant byte first) and > means big-endian. Read the wrong way, a record count of 300 turns into 738,263,040. That is not an error. It is a number, and it looks like a real one. The format specification decides the byte order, not the reader, and a parser that guesses is a parser that can be wrong without saying so.

Second, the printed bytes are misleading in their own way. That comma in the middle is not a comma in the data. Python shows any byte that happens to be printable ASCII as a character. 300 is 0x012C, stored little-endian as 2c 01, and 0x2C is the ASCII code for a comma. When I want to see the bytes as bytes, .hex() is more honest:

print(raw[6:10].hex(" "))
print(int.from_bytes(raw[6:10], "little"), int.from_bytes(raw[6:10], "big"))

2c 01 00 00
300 738263040

Third, the prefix also controls size, not just order:

print(struct.calcsize("<HI"), struct.calcsize("HI"))
try:
    struct.unpack("HI", raw[4:])
except struct.error as e:
    print("struct.error:", e)

6 8
struct.error: unpack requires a buffer of 8 bytes

With no prefix, struct uses the machine's native byte order and alignment, so it inserts padding between the 2-byte and 4-byte fields the way a C compiler would. With <, >, = or !, no padding is added (struct docs). The native size is a property of the machine running the script, not of the file. On my machine it was 8. For reading a file format, I now always write the byte order prefix explicitly.

What tripped me up: I assumed a wrong format string would fail loudly. Here it only failed because the length changed. Swap < for > and it happily returns nonsense.

Layer two: text mode is already an interpretation

Opening a file without b in the mode asks Python to decode the bytes into text, and that decoding does more than I expected:

import hashlib

with open("export.txt", "wb") as f:
    f.write(b"user=alice\r\nuser=bob\r\n")

with open("export.txt", "rb") as f:
    as_bytes = f.read()
with open("export.txt", "r", encoding="utf-8") as f:
    as_text = f.read()

print(len(as_bytes), len(as_text))
print(repr(as_text))
print(hashlib.sha256(as_bytes).hexdigest()[:16])
print(hashlib.sha256(as_text.encode("utf-8")).hexdigest()[:16])

22 20
'user=alice\nuser=bob\n'
336989692df6c4ba
5237f50ca4a04c3d

By default, text mode translates \r\n and \r line endings into \n on every platform, not just Windows (`open()` docs, the newline parameter). Two bytes vanished, and the hash changed. For reading content that is usually helpful. For verifying a file, it means the hash describes Python's interpretation, not the file. Hashing belongs on bytes opened with rb, or on hashlib.file_digest() (Python 3.11 and later), which takes a binary file object.

The encoding argument is part of the same decision. Without it, Python picks a default from the machine's locale, which can differ between machines, so I now pass it every time I open text. (PEP 686 makes UTF-8 the default starting with Python 3.15, but scripts still run on older versions.)

Layer three: a line is not a value

The most common way I went wrong at this layer was a comparison that should have matched and did not:

with open("known.txt", "w", encoding="utf-8") as f:
    f.write("9F86D081\n60303AE2\n")

found = {"9f86d081", "fd61a03a"}

with open("known.txt", encoding="utf-8") as f:
    raw_lines = f.readlines()
print(raw_lines)
print(found & set(raw_lines))

with open("known.txt", encoding="utf-8") as f:
    known = {line.strip().lower() for line in f if line.strip()}
print(found & known)

['9F86D081\n', '60303AE2\n']
set()
{'9f86d081'}

(The hashes are shortened to eight characters to keep things readable.)

readlines() keeps the newline at the end of each line, and hex digits can arrive in either case. The first comparison finds nothing, which in a hash lookup reads as "no known files here." That is the worst kind of wrong answer, because "no match" is a plausible result. Stripping whitespace, normalizing case and skipping blank lines turned it into the right one. The with block also matters: it closes the file even if something inside raises an error.

Layer four: a delimiter and a type are claims

Moving up to tables, pandas makes reading a whole export one call. It also makes two decisions for me unless I make them first. Here is a small tab-separated artifact list:

import pandas as pd

with open("artifacts.tsv", "w", encoding="utf-8") as f:
    f.write("item\tname\tsize\n"
            "0041\tnotes.txt\t512\n"
            "0042\tphoto.jpg\t20480\n"
            "0043\tcache.db\t4096\n"
            "0044\tinvoice.pdf\t20480\n")

wrong = pd.read_csv("artifacts.tsv")
print(wrong.shape)

df = pd.read_csv("artifacts.tsv", sep="\t")
print(df["item"].tolist())

df = pd.read_csv("artifacts.tsv", sep="\t", dtype={"item": str})
print(df["item"].tolist())

(4, 1)
[41, 42, 43, 44]
['0041', '0042', '0043', '0044']

With the default comma separator, every row became a single column. No error, just one wide column. With the right separator, pandas guessed that item was a number and dropped the leading zeros. If those are evidence item numbers, 41 no longer matches the 0041 on a label or in a report. Telling pandas the column is text kept the identifier as it was written. The file extension decides neither of these. I state the separator and the types of identifier columns myself.

Layer five: a label is not a position

This was the idea I most wanted to get right, because it is about citing a specific record. Every DataFrame row has an index label, and sorting keeps those labels attached:

by_size = df.sort_values("size", ascending=False, kind="stable")
print(by_size)
print(by_size.loc[0, "name"])
print(by_size.iloc[0]["name"])

   item         name   size
1  0042    photo.jpg  20480
3  0044  invoice.pdf  20480
2  0043     cache.db   4096
0  0041    notes.txt    512
notes.txt
photo.jpg

loc selects by label and iloc by position. Before sorting, they agree. After sorting, loc[0] still means "the row that was first in the file", while iloc[0] means "whatever is on top right now." Both are useful. They are not interchangeable, and mixing them up cites the wrong record with no warning. I like that the original labels survive a sort. They are a small bit of provenance: they say where each row came from.

Two details I had to look up. sort_values defaults to quicksort, and the pandas docs list only mergesort and stable as stable choices, so rows that tie (the two 20480-byte files) are not guaranteed to keep their file order. I pass kind="stable" when order among ties matters. And slicing differs between the two:

print(df.iloc[0:2]["name"].tolist())
print(df.loc[0:2, "name"].tolist())

['notes.txt', 'photo.jpg']
['notes.txt', 'photo.jpg', 'cache.db']

iloc slices like normal Python and stops before the end. loc includes both ends, which the pandas docs flag as "contrary to usual python slices."

Filters need the element-wise operators

try:
    df[df["size"] > 1000 and df["name"].str.endswith(".pdf")]
except ValueError as e:
    print("ValueError:", e)

big_pdfs = df[(df["size"] > 1000) & (df["name"].str.endswith(".pdf"))]
print(big_pdfs)
print(df[df["item"].isin(["0041", "0043"])].index.tolist())

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
   item         name   size
3  0044  invoice.pdf  20480
[0, 2]

Python's and wants one True or False, and a column of comparisons is many. The element-wise operators & and | combine them row by row, and each condition needs its own parentheses because & binds tighter than >. isin() handles "any of these values." The result keeps label 3, so the filtered row can still be traced back to the export.

One more quiet trap: the error message still suggests a.bool(), but pandas 3.0 removed Series.bool(). The other suggestions work. Even an error message is worth checking against the version you run.

A chart is just another file

The last layer goes the other way: from data back to a file.

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
ax.bar(df["item"], df["size"])
ax.set_title(f"Size by item (n={len(df)})")
ax.set_xlabel("Item")
ax.set_ylabel("Bytes")
fig.savefig("sizes.png")

with open("sizes.png", "rb") as f:
    print(f.read(8))

b'\x89PNG\r\n\x1a\n'

Two habits from this. The title takes its count from the data instead of a number I typed, so the chart cannot drift from the table behind it. And savefig with the non-interactive Agg backend writes the image without needing a screen, which suits scripts that run unattended. Reading the first eight bytes back shows the standard PNG signature, which is the same idea as the magic bytes at the start of this piece. A chart in a report is a file too, and it can be checked like one.

The pattern behind my mistakes

None of these were crashes. A wrong byte order produced a believable number. Text mode changed a hash. A newline turned a match into "no match." A default separator and a guessed type reshaped the table. A position was mistaken for a label. Each layer made a reasonable choice I had not made on purpose.

The takeaway I am keeping: every layer of reading is an interpretation. State each one explicitly, and keep a way back to the original bytes and the original row.

What I am applying

1. Write the byte order prefix (< or >) in every struct format string for file data.
2. Look at unfamiliar bytes with `.hex()`, not the default bytes display.
3. Hash files opened in binary mode, never decoded text.
4. Pass `encoding=` when opening text, and strip and normalize lines before comparing them.
5. State the separator and the type of identifier columns when loading a table.
6. Use `loc` to cite a record by label and `iloc` only for position, and sort with kind="stable" when ties matter.
7. Combine filters with `&` and `|`, each condition in parentheses.
8. Take chart labels and counts from the data, and save charts to files.

What this insight does and does not support

• Observed. Learning file handling, pandas and matplotlib in Python, my mistakes were silent misreadings at each layer: byte order, newline translation, unstripped lines, default separators, guessed types and label versus position. The examples and the file format are my own inventions and were run on Python 3.13 and 3.14 with pandas 3.0 and matplotlib 3.11.
• Provides context. Useful for anyone writing small scripts that read exports or binary headers, especially where results have to be traced back to the source. The behaviors described are documented in the Python and pandas docs linked above.
• Not established by this piece alone. These are one learner's notes, and I am still early in this. The container format is invented, the shortened hashes are illustrations, and none of this is a validated forensic method or a statement about any tool, product or organization.

Check your own scripts by what they actually read, not by what you meant them to read.

Views are my own.