<!-- Copyright (C) 2024 THE PACKAGE'S COPYRIGHT HOLDER -->

<h1>Determining which entries the user has already seen</h1>

<h2>Historical perspective</h2>

<p>Earlier versions of this program used the following approach to
determine which changlog or NEWS entries (hereafter "entries") are new
and should be displayed to the user:</p>

<ul>
<li>Group packages by source package.</li>
<li>Keep track of the highest version number of any of the packages in
the group and use that as the threshold for identifying new entries.</li>
<li>Display any entries with version numbers not less than the
previously determined version number.</li>
</ul>

<p>This approach was based on two assumptions, neither of which is always
true:</p>

<ol>
<li>Assume that the version numbering for all packages that come from
the same source package are in the same series.</li>
<li>Assume that the version numbering of entries always matches the
aforementioned version numbering.</li>
</ol>

<p>For an example of where these assumptions break down, look at the
dmsetup package:</p>

<ul>
<li>The source package for dmsetup is lvm2.</li>
<li>The version number for the dmsetup package is lower, but with a
higher epoch than, the version number for the lvm2 package.</li>
<li>The entries in changelog.Debian.gz use lvm2 version numbers, while
the ones in changelog.Debian.devmapper.gz use dmsetup version
numbers.</li>
</ul>

<p>This approach was also limited in that it only looked at
NEWS.Debian[.gz], changelog.Debian[.gz], changlog.Debian.arch[.gz],
and changelog[.gz]. For an example of where this fails, again look at
dmsetup, which has changelog.Debian.devmapper.gz.</p>

<p>Another technique used in earlier versions of this program was to
attempt heuristically to ignore version number suffixes which should
not be considered when evaluating whether a particular entry was new.
The employed heuristics were brittle, potentially leading to missed
entries or entries displayed multiple times.</p>

<h2>Current approach</h2>

<p>The current approach continues to use version numbers to assist in
determining which entries to display to users. However, it does so
more cautiously and in a more limited way which dramatically reduces
the likelihood of failing to show the user entries that they should
see.</p>

<p>Specifically:</p>

<ul>
<li>Version numbers are only considered when looking at changelog files,
not NEWS files.</li>
<li>Version numbers are used independently per package, not grouped by
source package.</li>
<li>Version numbers are only used for packages that already have an
earlier version installed when apt-listchanges is considering which
changelog entries to display for a new version.</li>
<li>The program must see both the new version's and the earlier
version's version number in a changelog before it will decide to
ignore the rest of the entries in that changelog.</li>
</ul>

<p>The final requirement above is only half-enforced if the package's
name matches a list of package patterns whose changelog version
numbers are trusted. In that case, we don't require seeing the new
version's package number before ignoring the rest; it's sufficient for
us to see a version number that is semantically less than the new
version. At the this paragraph was written, the only pattern on this
list is <code>linux-image-*</code>, which is necessary because the maintainers of
the signed kernel packages do funky things with the version numbers in
their changelog files which prevent the full test from working
properly.</p>

<p>In addition to version numbers, the current approach also uses
checksums of changelog and NEWS entries to determine which entries the
user has already seen and therefore does not need to see again.</p>

<p>For each entry, the program stores two checksums: a checksum of the
entire entry including its header (the line that contains the package
name, version number, suite, and urgency), content, and footer (the line
containing the maintainer and timestamp); and a second checksum of the
content and footer, with the header omitted.</p>

<p>Whenever the program sees an entry whose full checksum matches a
checksum already in the database, it stops parsing the NEWS or
changelog file at that point. Whenever the program sees an entry whose
content/footer checksum matches a checksum already in the database, it
omits that entry from what is displayed to the user but continues
parsing the file to see if any earlier entries should be displayed.</p>

<p>Caveat: none of the logic above applies when <code>--since</code>, <code>--latest</code>, or
<code>--show-all</code> are specified.</p>

<p>The database used by the current approach is significantly larger than
the database required for the historical approach -- a few megabytes
vs. a few kilobytes -- but it is still relatively small, and we
consider this an acceptable amount of space to use for a significantly
better-performing algorithm.</p>

<p>Because this approach uses entry checksums, it could theoretically
able entries from files like changelog.Debian.devmapper that the
historical approach ignored, though that functionality has not yet
been implemented.</p>

<h3>Edge case: no database, or no data for a package in the database</h3>

<p>When the persistent database is not being used in a particular
invocation of the program, or when there is no data for a particular
package in the database, then the above approach requires
modification.</p>

<p>In this case, we read and calculate checksums for the same package on
disk to seed the database before we parse the files in the package.</p>