I had to rescan a couple of networks. I got a lot of dupe messages. How can I remove the dupes?

--
yrNews Usenet Reader for iOS
http://appstore.com/yrNewsUsenetReader

--- Mystic BBS/NNTP v1.12 A35 (Linux/64)
* Origin: -=The ByteXchange BBS : bbs.thebytexchange.com=- (21:1/107)

On 10/03/17, Nugax said the following...

I had to rescan a couple of networks. I got a lot of dupe messages. How can I remove the dupes?

Before a rescan make sure to select all message area's for that network with the tab key then press / then select reset so they are all zero. Now do the rescan and wont have any dupes.

Hope this helps!

Cheers!
Pequito

--- Mystic BBS v1.12 A35 (Linux/32)
* Origin: Twinkle BBS # (21:1/126)

Yea I saw that after I asked. Thanks man!

On 08:17 03/10 , Pequito wrote:

On 10/03/17, Nugax said the following...

I had to rescan a couple of networks. I got a lot of dupe messages. How can I remove the dupes?

Before a rescan make sure to select all message area's for that network with >the tab key then press / then select reset so they are all zero. Now do the >rescan and wont have any dupes.

Hope this helps!

Cheers!
Pequito

--- Mystic BBS v1.12 A35 (Linux/32)
* Origin: Twinkle BBS # (21:1/126)

--
yrNews Usenet Reader for iOS
http://appstore.com/yrNewsUsenetReader

--- Mystic BBS/NNTP v1.12 A35 (Linux/64)
* Origin: -=The ByteXchange BBS : bbs.thebytexchange.com=- (21:1/107)

On 10/03/17, Nugax said the following...

I had to rescan a couple of networks. I got a lot of dupe messages. How can I remove the dupes?

It's easiest to just not make dupes. Either delete everything first and then %RESCAN or %RESCAN D=5 when you know you missed 5 days. That second one is
what I do to keep dupes down. I RESCAN for the number of days I was down or off.
Once you have dupes one thing you can do is limit the number of messages you save to cut off the older ones. mutil cutdupes.ini (or whatever you want to call it) can be used to limit the number of messages in an area to the newest 1000 or whatever. If you have 2000 messages and 1000 are dupes, just limit
that area to 1000 messages (Edit the message base) and use mutil to PurgeMessageBases to cut them. Then go back and change the limit after you PurgeMessageBases to cut them (back up to 2000 in this case).

--- Mystic BBS v1.12 A35 (Windows/64)
* Origin: BetaTest bcw142.zapto.org:2323 (21:1/145.4)

On 10/03/17, Pequito pondered and said...

Before a rescan make sure to select all message area's for that network with the tab key then press / then select reset so they are all zero.
Now do the rescan and wont have any dupes.

Interestingly my experiences have been doing a message base reset after
tossing a bunch of messages in followed by another rescan... has led to many messages being rejected as dupes still. Why? Because even if you reset your
JAM bases prior to trying another mass rescan of message in from a HUB your Mystic system still retains records of all the past messages tossed prior to the base reset.

It does this in the file called echodupes.dat found in your data directory.
So if you're happy to blow out all out past dupe checking history by deleting this file then when you next retoss messages in from a HUB doing a full
rescan (having first reset all your bases) then all messages will toss in
fine.

** warning - you're mileage may vary and deleting this .dat file is NOT recommended **

Best, Paul

--- Mystic BBS v1.12 A35 (Windows/32)
* Origin: Agency BBS | telnet://agency.bbs.geek.nz (21:1/101)

Interestingly my experiences have been doing a message base reset after tossing a bunch of messages in followed by another rescan... has led to many messages being rejected as dupes still. Why? Because even if you

When you reset message bases (using /R in the editor) Mystic will ask you if you also want to reset the duplicate message detectors, so you shouldn't ever have to physically delete any files manually.

--- Mystic BBS v1.12 A36 (Windows/32)
* Origin: Sector 7 [Mystic BBS WHQ] (21:1/108)

On 10/22/17, g00r00 pondered and said...

When you reset message bases (using /R in the editor) Mystic will ask
you if you also want to reset the duplicate message detectors, so you shouldn't ever have to physically delete any files manually.

OK will test again next time I need to go down this path, in the past I had
to nuke the .dat file as the reset didn't seem to do enough. :)

--- Mystic BBS v1.12 A35 (Windows/32)
* Origin: Agency BBS | telnet://agency.bbs.geek.nz (21:1/101)

When you reset message bases (using /R in the editor) Mystic will ask you if you also want to reset the duplicate message detectors, so you shouldn't ever have to physically delete any files manually.

OK will test again next time I need to go down this path, in the past I had to nuke the .dat file as the reset didn't seem to do enough. :)

I see. I suppose there could be a conflict with this and having echomail running in the background. I don't think I have anything in place to detect
if the tossers are in use when it asks that question.

--- Mystic BBS v1.12 A36 (Windows/32)
* Origin: Sector 7 [Mystic BBS WHQ] (21:1/108)

On 10/22/17, g00r00 pondered and said...

OK will test again next time I need to go down this path, in the past had to nuke the .dat file as the reset didn't seem to do enough. :)

I see. I suppose there could be a conflict with this and having echomail running in the background. I don't think I have anything in place to detect if the tossers are in use when it asks that question.

The issue was (at the time) whilst a reset wiped the messages displayed to
the user from the base it did not appear to clear their imported records from the echodupe.dat file. So a full rescan of the messages from the HUB and subsequent import of them back in to the BBS failed as they were seen a dupes still by the BBS. So that's why I had been removing the .dat file. I hasten
to add this kind of thing was only taking place for me when I helped to set
up a new node and they were setting up a new system without any other
networks attached etc. so it was in that early 3-7 days of out of the box set up and I was just trying to ensure the new node had a full back catalog of fsxNet messages :)

Best, Paul

--- Mystic BBS v1.12 A35 (Windows/32)
* Origin: Agency BBS | telnet://agency.bbs.geek.nz (21:1/101)

The issue was (at the time) whilst a reset wiped the messages displayed
to the user from the base it did not appear to clear their imported records from the echodupe.dat file. So a full rescan of the messages

Right, it doesn't unless you answer "Yes" to the "Reset the dupe database?" question that it should be asking you when you do a /R.

But I am concerned that if the tosser is running when this happens that it could cause unknown results.

--- Mystic BBS v1.12 A36 (Windows/32)
* Origin: Sector 7 [Mystic BBS WHQ] (21:1/108)

On 10/23/17, g00r00 pondered and said...

Right, it doesn't unless you answer "Yes" to the "Reset the dupe database?" question that it should be asking you when you do a /R.

I'm going to need to set up a test system and test this again :)

But I am concerned that if the tosser is running when this happens that
it could cause unknown results.

Yep fair enough that could end badly.

--- Mystic BBS v1.12 A35 (Windows/32)
* Origin: Agency BBS | telnet://agency.bbs.geek.nz (21:1/101)