Fossil SCM
Improve the recommented value for robot-exception.
Commit
3bbd18a284c8bd6a8c1d46a6faa7ee33142e5fec53bb64792aab53d3c84e57fd
Parent
57c8bec8169e072…
1 file changed
+3
-2
+3
-2
| --- src/setup.c | ||
| +++ src/setup.c | ||
| @@ -501,17 +501,18 @@ | ||
| 501 | 501 | textarea_attribute("", 2, 80, |
| 502 | 502 | "robot-restrict", "rbrestrict", robot_restrict_default(), 0); |
| 503 | 503 | |
| 504 | 504 | @ <hr> |
| 505 | 505 | @ <p><b>Exceptions to anti-robot restrictions</b><br> |
| 506 | - @ The entry below is a list of regular expressions, one per line. | |
| 506 | + @ The entry below is a list of | |
| 507 | + @ <a href="%R/re_rules">regular expressions</a>, one per line. | |
| 507 | 508 | @ If any of these regular expressions match the input URL, then the |
| 508 | 509 | @ request is exempt from anti-robot defenses. Use this, for example, |
| 509 | 510 | @ to allow scripts to download release tarballs using a pattern |
| 510 | 511 | @ like:</p> |
| 511 | 512 | @ <p> |
| 512 | - @   <tt>^/tarball\b.*\b(version-|release)\b</tt> | |
| 513 | + @   <tt>^/tarball/(version-[0-9.]+|release)/</tt> | |
| 513 | 514 | @ <p>The pattern should match against the REQUEST_URI with the |
| 514 | 515 | @ SCRIPT_NAME prefix removed, and with QUERY_STRING appended following |
| 515 | 516 | @ a "?" if QUERY_STRING exists. (Property: robot-exception)<br> |
| 516 | 517 | textarea_attribute("", 3, 80, |
| 517 | 518 | "robot-exception", "rbexcept", "", 0); |
| 518 | 519 |
| --- src/setup.c | |
| +++ src/setup.c | |
| @@ -501,17 +501,18 @@ | |
| 501 | textarea_attribute("", 2, 80, |
| 502 | "robot-restrict", "rbrestrict", robot_restrict_default(), 0); |
| 503 | |
| 504 | @ <hr> |
| 505 | @ <p><b>Exceptions to anti-robot restrictions</b><br> |
| 506 | @ The entry below is a list of regular expressions, one per line. |
| 507 | @ If any of these regular expressions match the input URL, then the |
| 508 | @ request is exempt from anti-robot defenses. Use this, for example, |
| 509 | @ to allow scripts to download release tarballs using a pattern |
| 510 | @ like:</p> |
| 511 | @ <p> |
| 512 | @   <tt>^/tarball\b.*\b(version-|release)\b</tt> |
| 513 | @ <p>The pattern should match against the REQUEST_URI with the |
| 514 | @ SCRIPT_NAME prefix removed, and with QUERY_STRING appended following |
| 515 | @ a "?" if QUERY_STRING exists. (Property: robot-exception)<br> |
| 516 | textarea_attribute("", 3, 80, |
| 517 | "robot-exception", "rbexcept", "", 0); |
| 518 |
| --- src/setup.c | |
| +++ src/setup.c | |
| @@ -501,17 +501,18 @@ | |
| 501 | textarea_attribute("", 2, 80, |
| 502 | "robot-restrict", "rbrestrict", robot_restrict_default(), 0); |
| 503 | |
| 504 | @ <hr> |
| 505 | @ <p><b>Exceptions to anti-robot restrictions</b><br> |
| 506 | @ The entry below is a list of |
| 507 | @ <a href="%R/re_rules">regular expressions</a>, one per line. |
| 508 | @ If any of these regular expressions match the input URL, then the |
| 509 | @ request is exempt from anti-robot defenses. Use this, for example, |
| 510 | @ to allow scripts to download release tarballs using a pattern |
| 511 | @ like:</p> |
| 512 | @ <p> |
| 513 | @   <tt>^/tarball/(version-[0-9.]+|release)/</tt> |
| 514 | @ <p>The pattern should match against the REQUEST_URI with the |
| 515 | @ SCRIPT_NAME prefix removed, and with QUERY_STRING appended following |
| 516 | @ a "?" if QUERY_STRING exists. (Property: robot-exception)<br> |
| 517 | textarea_attribute("", 3, 80, |
| 518 | "robot-exception", "rbexcept", "", 0); |
| 519 |