1. Forum
  2. Usenet Intenat
  3. ALT.BBS.SYNCHRONET

  • webv4 and web crawlers

    From Al@1:229/2 to All on Monday, August 10, 2020 13:12:31
    From: al@TRMB.remove-kc1-this

    To: echicken
    Hello echicken,

    I have been messing around with the stock web interface and webv4 and I am just
    wondering if there is a way to block web crawlers from the message areas the way the stock interface does it with the robots.txt?

    Ttyl :-),
    Al

    ... Cursor: An expert in four-letter words

    ---
    � Synchronet � The Rusty MailBox - Penticton, BC Canada
    --- Synchronet 3.18a-Win32 NewsLink 1.113
    * Vertrauen - Riverside County, California - telnet://vert.synchro.net

    --- SoupGate-Win32 v1.05
    * Origin: www.darkrealms.ca (1:229/2)
  • From echicken@1:229/2 to All on Monday, August 10, 2020 16:39:24
    From: echicken@ECBBS.remove-656-this

    To: Al
    Re: webv4 and web crawlers
    By: Al to echicken on Mon Aug 10 2020 13:12:31

    I have been messing around with the stock web interface and webv4 and I am
    just wondering if there
    is a way to block web crawlers from the message areas the way the stock
    interface does it with the
    robots.txt?

    You can put a robots.txt file inside the document root (as on any webserver); I
    don't remember the exact path offhand, but the directory name is 'root' and it has an 'index.xjs' file in it.

    You may also want to configure a web 'guest' user and limit its access to message areas, giving unauthenticated bots less surface area to crawl over. (In
    modopts.ini->[web], set 'guest' to the username of whatever account you want guests to be treated as.
    )

    ---
    echicken
    electronic chicken bbs - bbs.electronicchicken.com
    � Synchronet � electronic chicken bbs - bbs.electronicchicken.com
    --- Synchronet 3.18a-Win32 NewsLink 1.113
    * Vertrauen - Riverside County, California - telnet://vert.synchro.net

    --- SoupGate-Win32 v1.05
    * Origin: www.darkrealms.ca (1:229/2)