Why CSV Is Harder Than It Looks

It looks trivial and it is not. There is no single CSV standard, so real files differ in separator, quoting, encoding and line endings, and every one of those differences breaks naive parsing.

Use a CSV library. Splitting on commas works until the first quoted field containing a comma, which in real data is usually within the first few hundred rows.

EXAMPLE
loading editor…

Encoding is the second trap. A file full of strange characters is almost always latin-1 or cp1252 rather than UTF-8. A first column mysteriously named id is a byte order mark; open it with utf-8-sig and it disappears.

In Python, newline="" on the open call is not optional. Without it, quoted fields containing line breaks are corrupted, silently.

Mark this lesson complete

Signed in, your progress follows you to every device.

Next: Everything Is Text Until You Convert It →
← Back