From: nospam.Jay.Harris@f664.n229.z1.fidonet.org

Hi g00r00,

I've started seeing some duplicates in my Fidonet bases. I have three Fidonet hubs, all roughly with the same bases available (with the odd exception of some local/regional echos only available on one or the other).

I have these two lines set in my mailin.ini:

;dupe_msg_index = -1
dupe_db_size = 250000

Here is an example of a dupe that made it into my FIDOTEST base:

Via 1:229/426: https://paste.ee/p/u0Us9
Via 3:770/1: https://paste.ee/p/GIS7F

All of the kludges are the same with the exception of the seen-by and path lines. There does seem to be a difference in the message body where the quoted seen-by's have been stripped out in the message I received via 1:229/426.


Jay

... I call things as I see them; If I didn't see them, I make them up!

--- SoupGate-Win32 v1.05
* Origin: www.darkrealms.ca (1:229/2)

From: nospam.Paul.Hayton@f100.n770.z3.fidonet.org

On 11 Oct 2021 at 08:20p, Jay Harris pondered and said...

I've started seeing some duplicates in my Fidonet bases.

Here are two more than came through today. I've turned off my feeds to 2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426:

Hi Jay

I run the 3:770/1 HUB as you know.

Not sure if this is a Mystic thing. But to start with the two links below

https://paste.ee/p/hBhCR
https://paste.ee/p/nN9aV

neither contain 3:770/1 in the PATH line so in those examples the system is not depicted. If you did get a packet direct from 3:770/1 it would show as the last system in your PATH before your system imported it.

What I can see is in the two examples above the common systems in both PATH lines are 261/38 and 229/426 so I would start with those systems to check if any modifications occurred there.

Hope that helps.

Best, Paul

--- SoupGate-Win32 v1.05
* Origin: www.darkrealms.ca (1:229/2)

From: nospam.Al@p4.f757.n153.z1.fidonet.org

Hello Jay,

Here are two more than came through today. I've turned off my feeds
to 2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426:

https://paste.ee/p/hBhCR

https://paste.ee/p/nN9aV

Again, the kludges look the same (except for the path and seen-by
lines) but the message body has again been modified. One message has
a SLMR line where the other one doesn't.

I don't think that is a mystic issue, something is happening to messages arriving at your node. The message body should never be modified or altered in any way or dupes could happen.

I only have one copy of those messages here that include the SLMR line.

Ttyl :-),
Al

... When you learn the answers, they change the questions.

--- SoupGate-Win32 v1.05
* Origin: www.darkrealms.ca (1:229/2)

From: nospam.Jay.Harris@f664.n229.z1.fidonet.org

On 10 Oct 2021, Jay Harris said the following...

I've started seeing some duplicates in my Fidonet bases.

Here are two more than came through today. I've turned off my feeds to
2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426:

https://paste.ee/p/hBhCR

https://paste.ee/p/nN9aV

Again, the kludges look the same (except for the path and seen-by lines) but the message body has again been modified. One message has a SLMR line where the other one doesn't.


Jay

... It takes a long time to understand nothing.

--- SoupGate-Win32 v1.05
* Origin: www.darkrealms.ca (1:229/2)

From: nospam.Jay.Harris@f664.n229.z1.fidonet.org

On 11 Oct 2021, Al said the following...

I don't think that is a mystic issue, something is happening to messages arriving at your node.

The main reason I'm reporting it here is because Mystic doesn't seem to be catching these dupes. The MSGID & date/time are the same which to my mind means the 2nd one should be considered a dupe, even if the body of the message isn't exactly (but mostly) the same.


Jay

... Assassination is the extreme form of censorship.

--- SoupGate-Win32 v1.05
* Origin: www.darkrealms.ca (1:229/2)

From: nospam.Jay.Harris@f664.n229.z1.fidonet.org

On 12 Oct 2021, Paul Hayton said the following...

Here are two more than came through today. I've turned off my feeds 2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426

neither contain 3:770/1 in the PATH line so in those examples the system is not depicted. If you did get a packet direct from 3:770/1 it would
show as the last system in your PATH before your system imported it.

Yup, as mentioned above I've unsubscribed from all echos on 2:221/6 & 3:770/1 while I try and figure out where this is happening. I've been in touch with Nick who says he doesn't believe the issue is at his system.

What I can see is in the two examples above the common systems in both PATH lines are 261/38 and 229/426 so I would start with those systems to check if any modifications occurred there.

Yup, I've also noticed 301/1 is another common node in the path of the messages where the body has been modified.


Jay

... WARNING! Removal of this tagline prohibited by law!

--- SoupGate-Win32 v1.05
* Origin: www.darkrealms.ca (1:229/2)

From: nospam.deon@f1.n770.z2602.fidonet.org

Re: Re: Dupes?
By: Jay Harris to g00r00 on Mon Oct 11 2021 08:20 pm

Here are two more than came through today. I've turned off my feeds to 2:221/6 & 3:770/1 for now and am just hooked up to 1:229/426: https://paste.ee/p/hBhCR
https://paste.ee/p/nN9aV
Again, the kludges look the same (except for the path and seen-by lines) but the message body has again been modified. One message has a SLMR line where
other one doesn't.

Howdy,

For me it looks like three things:

* Your BBS didnt consider it a dupe

* 229/426 didnt consider it a dupe - since it sent it to you twice.

* Somebody after 261/38 modified the message - possibly 301/1, 320/219 or 240/5832, since it looks like 261/38 sent it 2 ways, that end up at 229/426. So either 301/1 removed the bottom 2 lines, or one of 320/219, 240/5832 added them.

This could go 2 ways, depending if the authors of the software involved think they are doing the "right thing" or not.

1) A consistent method to identify a duplicate - which there is an FTS document that can help here,
2) "Thou should not modify message content" - dont recall if that is an FTS document somewhere

Fortunately for you, there is some software that handles this situation well, so even if something upsream is not doing what is expected you can mitigate it yourself.

Getting popcorn...


...ëîåï

--- SoupGate-Win32 v1.05
* Origin: www.darkrealms.ca (1:229/2)

From: nospam.Nick.Andre@f426.n229.z1.fidonet.org

On 13 Oct 21 17:23:13, G00R00 said the following to Jay Harris:

Cool, thanks for the clarification g00r00. Hopefully we've tracked dow the node that is modifying the messages. I believe I also saw Nick released an update to D'Bridge that may also help. (Thanks Nick!)

Great to hear!

I have "More Cowbell" in my code than you do...

All jokes aside - D'Bridge has a hook to play a batch-file when personal mail is received and comes with a sample MP3 player and the voice of HAL from 2001.

Does Mystic have the same hook to call a program on personal mail tossed?

Also... The JAM storage - I assume Mystic keeps "last read" pointers for each user in a seperate file outside of the spec? Thats what I'm trying to get my head around with. The spec seems to allow this but its... iffy.

Nick

--- SoupGate-Win32 v1.05
* Origin: www.darkrealms.ca (1:229/2)