+++ /dev/null
-
- BOOKMARKS database and internet robot
-
- Here is a set of classes, libraries and programs I use to manipulate my
-bookmarks.html. I like Netscape Navigator, but I need more features, so I am
-writing these programs for my needs. I need to extend Navigator's "What's new"
-feature (Navigator 4 named it "Update bookmarks").
-
- These programs are intended to run as follows.
-1. bkmk2db converts bookmarks.html to bookmarks.db.
-2. chk_urls (Internet robot) runs against bookmarks.db, checks every URL and
- saves results in check.db.
-3. db2bkmk converts bookmarks.db back to bookmarks.html.
- Then I use this bookmarks file and...
-4. bkmk2db converts bookmarks.html to bookmarks.db.
-5. chk_urls (Internet robot) runs against bookmarks.db, checks every URL and
- saves results in check.db (old file copied to check.old).
-6. (An yet unnamed program) will compare check.old with check.db and generate
-detailed report. For example:
- this URL is unchanged
- this URL is changed
- this URL is unavailable due to: host not found...
-
- Bookmarks database programs are almost debugged. What need to be done is
-support for aliases. Second version of the internet robot is finished.
-
- Although not required, these programs work fine with tty_pbar.py (my little
-module for creating text-mode progress bars).
-
-COPYRIGHT and LEGAL ISSUES
- All programs copyrighted by Oleg Broytmann and PhiloSoft Design. All
-sources protected by GNU GPL. Programs are provided "as-is", without any kind
-of warranty. All usual blah-blah-blah.
-
- #include <disclaimer>
-
-
------------------------------- bkmk2db ------------------------------
- NAME
- bkmk2db.py - script to convert bookmarks.html to FLAD database.
-
- SYNOPSIS
- bkmk2db.py [-its] [/path/to/bookmarks.html]
-
- DESCRIPTION
- bkmk2db.py splits given file (or ./bookmarks.html) into FLAD database
- bookmarks.db in current directory.
-
- Options:
- -i
- Inhibit progress bar. Default is to display progress bar if
- stderr.isatty()
-
- -t
- Convert to text file (for debugging). Default is to convert to
- FLAD.
-
- -s
- Suppress output of statistics at the end of the program. Default
- is to write how many lines the program read and how many URLs
- parsed. Also suppress some messages during run.
-
- BUGS
- The program starts working by writing lines to header file until
- BookmarksParser initializes its own output file (this occur when
- parser encountered 1st <DL> tag). It is misdesign.
-
- Empty comments (no text after <DD>) are not marked specially in
- database, so db2bkmk.py will not reconstruct it. I don't need empty
- <DD>s, so I consider it as feature, not a real bug.
-
- Aliases are not supported (yet).
-
-
------------------------------- db2bkmk ------------------------------
- NAME
- db2bkmk.py - script to reconstruct bookmarks.html back from FLAD
- database.
-
- SYNOPSIS
- db2bkmk.py [-is] [-t dict.db [-r]]
-
- DESCRIPTION
- db2bkmk.py reads bookmarks.db and creates two HTML files -
- public.html and private.html. The latter is just full
- bookmarks.html, while the former file hides private folder.
-
- Options:
- -i
- Inhibit progress bar. Default is to display progress bar if
- stderr.isatty()
-
- -s
- Suppress output of statistics at the end of the program. Default is
- to write how many records the program proceed and how many URLs
- created. Also suppress some messages during run.
-
- -t dict.db
- For most tasks, if someone need to process bookmarks.db in a
- regular way (for example, replace all "gopher://gopher." with
- "http://www."), it is easy to write special program, processing
- every DB record. For some tasks it is even simpler and faster to
- write sed/awk scripts. But there are cases when someone need to
- process bookmarks.db in a non-regular way: one URL must be changed
- in one way, another URL - in second way, etc. The -t option allows
- to use external dictionary for such translation. The dictionary
- itself is again FLAD database, where every record have two keys -
- URL1 and URL2. With -t option in effect, db2bkmk generates
- {private,public}.html, renames them to {private,public}.1, and
- then translates the entire bookmarks.db again, generating
- {private,public}.2 (totally 4 files), where every URL1 replaced
- with URL2 from dictionary. (See koi2win.db for example of
- translation dictionary)
-
- -r
- Reverse the effect of -t option - translate from URL2 to URL1.
-
- BUGS
- There are three hacks under line marked with "Dirty hacks here":
- 1. if record["Folder"] == "Private links":
- This is to hide passwords from my bookmarks file.
-
- 2. if record["Folder"] == "All the rest - Unclassified":
- outfile.write(" "*level + "<DT><H3 NEWITEMHEADER ...")
- First, I compare folder name with fixed string. This is real string
- from my bookmarks.html. If anyone want to use the program (s)he
- should change at least the very strings "Private links" and "All the
- rest - Unclassified". Second, I use netscapism "NEWITEMHEADER".
- Yes, I wrote these programs for Navigator's bookmarks.html, but I
- still would not like to use too many netscapisms here.
-
-
------------------------------- check_db ------------------------------
- NAME
- check_db.py - script to test generated FLAD database.
-
- SYNOPSIS
- check_db.py [-s] [-l logfile.err]
-
- DESCRIPTION
- check_db.py reads bookmarks.db and tests for various conditions and
- possible errors. Current tests are for duplicated URLs and too big
- indent. "Indent without folder" or "Indent too big" may occur if
- someone edit bookmarks.db manually, inserting a record with incorrect
- (higher) level (lower levels indents are ok). Every record tested for
- correct format (that there are no spare keys, date formats are
- correct).
-
- Options:
- -l logfile.err
- Put error log into log file (errors are printed to stderr
- anyway).
-
- -s
- Suppress information messages while running (errors are printed
- anyway).
-
-
------------------------------- chk_urls -----------------------------
- NAME
- chk_urls.py - Internet robot
-
- SYNOPSIS
- chk_urls.py [-is]
-
- DESCRIPTION
- chk_urls.py runs against bookmarks.db, checking every URL and store
- results in check.db. check.db is FLAD database almost identical to
- bookmarks.db, with modified LastVisit/LastModified fields. Additional
- field Error appeared in records that have not been checked by some
- reasons; the reason is a content of Error field.
- After every 100 URLs chk_urls creates checkpoint file check.dat (in
- set_checkpoint()). The file is FLAD suitable to pass to
- fladc.load_file() (in get_checkpoint()). If interrupted by ^C, killed
- or crashed, chk_urls can be restarted, and checkpoint file helps to
- restart from interrupted state. Checkpoint stores size and mtime of
- bookmarks.db (to note if the file changed while chk_urls interrupted)
- and last checked record. If chk_urls cannot find checkpoint file, or
- bookmarks.html changed, chk_urls will restart from the beginning. If
- there is valid checkpoint and size/mtime are ok - chk_urls will start
- from interrupted record.
-
- Options:
- -i
- Inhibit progress bar. Default is to display progress bar if
- stderr.isatty()
-
- -s
- Suppress output of statistics at the end of the program. Default is
- to write how many records the program proceed and how many URLs
- checked. Also suppress some messages during run.
-
- BUGS
- Ugly mechanism to catch welcome message from FTP server (from urllib).
-
-
------------------------------- chk_urls -----------------------------
- NAME
- check_urls2.py - Internet robot
-
- SYNOPSIS
- check_urls2.py [-is]
-
- DESCRIPTION
- check_urls2 is just a second version of chk_urls.py. It forks off a child
- process and the child check URLs. The parent monitors the child and kills
- it, if there is no answer within 15 minutes.
projects / bookmarks_db.git / blobdiff