F40 Change Proposal: Switch bogofilter to use SQLite (Self-Contained)

Project Discussion Change Proposals
,
1

Switch bogofilter to use SQLite

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

:link: Summary

Switch bogofilter to use SQLite as its database engine, rather than Berkeley DB (libdb).

:link: Owner

:link: Detailed Description

Switch bogofilter to use SQLite as its database engine, rather than Berkeley DB (libdb). Another change (Changes/Libdb deprecated - Fedora Project Wiki) marked libdb as deprecated, and that change lists bogofilter as a dependency. Thus this change fixes another application to avoid the deprecated libdb.

:link: Benefit to Fedora[edit]

Fedora will have one less dependency on the deprecated libdb package. Additionally, other distributions have already migrated to SQLite, and this will allow sharing word lists with those distributions. For example, perhaps a workstation running Fedora generates wordlist.db before installing it on another computer running Alpine and acting as a server.

:link: Scope[edit]

  • Proposal owners:

Merge pull request PR#2: Resolves: #1788486 (Switch to SQLite database engine) - rpms/bogofilter - src.fedoraproject.org. This makes the database backend conditional, with SQLite being the default. Support for libdb can be conditionally compiled to create a migration tool capable of migrating existing libdb databases to SQLite.

  • Other developers: N/A (not needed for this Change)
  • Release engineering: N/A (not needed for this Change)
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Community Initiatives: Support Mark libdb as deprecated

:link: Upgrade/compatibility impact

Bogofilter can support only one database backend at a time, and thus a new SQLite bogofilter package will be unable to process libdb data. Thus the new package provides a migration script.

:link: How To Test

This test generates a word list and migrates it to work with the new SQLite backend. Install original bogofilter and add at least one word to its database, for example with:

echo abc | bogofilter --bogofilter-dir=/tmp/bftest/ --register-spam

Bogofilter will create the directory /tmp/bftest/, and it will contain a wordslist.db file. To verify the word had been added run:

bogoutil -d /tmp/bftest/wordlist.db

Install the updated bogofilter and migrate the existing libdb database with:

bogomigrate-berkeley /tmp/bftest/wordlist.db

This tool will print whether the migration succeeded. Verify the “abc” word is present in the newly created SQLite database with:

bogoutil -d /tmp/bftest/wordlist.db

:link: User Experience

:link: Dependencies

N/A (not needed for this Change)

:link: Contingency Plan

  • Contingency mechanism: N/A (not a System Wide Change)
  • Contingency deadline: N/A (not a System Wide Change)
  • Blocks release? No

:link: Documentation

N/A (not a System Wide Change)

:link: Release Notes

The bogofilter package switched its database engine from Berkeley DB (libdb) to SQLite because Fedora deprecated libdb. Users can migrate their word lists manually with bogomigrate-berkeley ~/.bogofiler/wordlist.db.

1 Like
2

This change proposal has now been submitted to FESCo for voting and this topic thread will now be closed.

Ticket #3091

For details on the next phase of the process, visit our Changes Policy documentation.

5

Using my new-found powers for good, and after a good suggestion from a friend who has been involved in the Fedora project for a long, long time, ( :grin:) I will leave the topics open until their respective deadlines - see F40 schedule for exact dates.

I will use the date for the System-Wide changes to close topics related to this type of change, and same for changes tagged as Self-Contained.

Sorry for the noise and I hope this works out quite well for us :slight_smile:
Aoife

6

Bogofilter can support only one database backend at a time, and thus a new SQLite bogofilter package will be unable to process libdb data. Thus the new package provides a migration script.

OK, but what happens if the user doesn’t do the migration? Is there some clear message, data loss, etc?

Could we make it so that the conversion happens automatically?

7

Bogofilter claims an error about the database format in such case. The migration tool cannot be run automatically, that would require either changes in the upstream code, or write wrapper scripts to each bogofilter command and run the migration tool, if needed. The tool is compiled conditionally in the .spec file, thus it might not be always available.

In fact, I did think of migrating the database on the package update in users’ homes, but then I moved away from that idea, because it’s not good to touch users’ homes under their hands. I think I’ve been told so as well.

8

Yes, doing that during package install is explicitly forbidden by the packaging guidelines. See No Files or Directories Under /srv, /usr/local, or /home/$USER.

9

I’ve been using evolution with evolution-bogofilter for quite some time now, and it recently just stopped working. The plugin did not show me any errors, but I noticed that Spam was not being filtered and if I looked at my PS there would be many <defunct> child bogofilter threads.

I was able to manually migrate my wordlist.db and this alleviated the issue.

I had no idea the Berkeley DB -to- SQLite migration was required until after I’d installed the debuginfo packages and ran evolution with strace. Only then did I see any error.

The error from strace was the following.

bogofilter[4631]: Error preparing "SELECT name FROM sqlite_master WHERE type='table' AND name='bogofilter';": file is not a database (#26)_
bogofilter[4631]: Error on database /home/p/.bogofilter/wordlist.db: file is not a database_
10

I am not sure how best to handle this. The new bogofilter package provides a script, bogomigrate-berkeley, along with instructions on how to use it in the package description. The change proposal provides a release note that says the same thing. Note that packages are forbidden from modifying $HOME in an automated way, such as to automatically migrate a bogofilter database.

Perhaps this warrants bug requests/requests for enhancement in evolution-bogofilter and claws-mail-plugins-bogofilter suggesting the plugins be made more forthright about the problem. An error message would go a long way to avoid having to use strace to diagnose.

11

I agree with Michael, there is not much more what could be done without playing with user files, which is a bad idea, not talking it’s forbidden.

I tested this in rawhide with:

bogofilter-1.2.5-13.fc40.x86_64
evolution-3.51.1-1.fc40.x86_64

and having there the libdb database. I marked a message as not being Junk in Evolution, which resulted in an error in the GUI:

Error while Filtering folder “Home : INBOX”.

Bogofilter either crashed or failed to process a mail message

and as I ran evolution from a terminal, I see there is printed:

bogofilter[3468]: Error preparing "SELECT name FROM sqlite_master WHERE type='table' AND name='bogofilter';": file is not a database (#26)_
bogofilter[3468]: Error on database /home/zyx/.bogofilter/wordlist.db: file is not a database_
Can't open file 'wordlist.db' in directory '/home/zyx/.bogofilter'.
error #2 - No such file or directory.

Remember to register some spam and ham messages before you
use bogofilter to evaluate mail for its probable spam status!

(evolution:3074): module-bogofilter-WARNING **: 12:12:04.982: Bogofilter: Unexpected exit code (3) while registering ham

All the text, except of the last line, is from the bogofilter. It’s very techy and probably not easy to understand what to do, I agree with that, still I’m not sure what else could be done.

The error #2 - No such file or directory. is misleading on its own, when bogofilter claims file is not a database, but that’s an unrelated thing.

12

I am familiar with the GUI error area in Evolution and I often get messages there from time to time, I definitely did not get any bogofilter-related message. I do not know why.

I did not think to run Evolution from a terminal right away, that would have been a good troubleshooting step (I do this sometimes as well but it did not occur to me.) Running Evolution with strace I think effectively did this.

13

your command have one typo : File ‘/home/sergio/.bogofiler/wordlist.db’ does not exist

bogomigrate-berkeley ~/.bogofilter/wordlist.db
Successfully migrated ‘~/.bogofilter/wordlist.db’ with 89576 entries.
Backup of the original file is stored as ‘~/.bogofilter/wordlist.db.berkeley.bak’.

and fix the errors:
evolution[1294881]: bogofilter[1294881]: Error preparing “SELECT name FROM sqlite_master WHERE type=‘table’ AND name=‘bogofilter’;”: file is not a database (#26)
evolution[1294881]: bogofilter[1294881]: Error on database /home/sergio/.bogofilter/wordlist.db: file is not a database_
evolution[1294881]: Can’t open file ‘wordlist.db’ in directory ‘/home/sergio/.bogofilter’.

14

How do you feel about the proposal as written?

  • Strongly in favor
  • In favor, with reservations
  • Neutral
  • Opposed, but could be convinced
  • Strongly opposed
0 voters

If you are in favor but have reservations, or are opposed but something could change your mind, please explain in a reply.

We want everyone to be heard, but many posts repeating the same thing actually makes that harder. If you have something new to say, please say it. If, instead, you find someone has already covered what you’d like to express, please simply give that post a :heart: instead of reiterating. You can even do this by email, by replying with the heart emoji or just “+1”. This will make long topics easier to follow.

Please note that this is an advisory “straw poll” meant to gauge sentiment. It isn’t a vote or a scientific survey. See About the Change Proposals category for more about the Change Process and moderation policy.

15

Hi,
thanks for reporting, though it’s odd. The change proposal page has it
fixed, see:
https://fedoraproject.org/wiki/Changes/BogofilterSqlite#Release_Notes
and the final release notes for the Fedora 40 has it correctly too, see:

it’s only wrong here, in the description, which had been created from
the proposal page in time of filling it. The description here does not
seem to be updated on the wiki page changes (which is for good and bad
at the same time).

Bye,
Milan
16

Closing change proposals for F40. The ship has sailed. Of course we can still discuss if needed, but please open new topics for that.