Checks/synchs a local package mirror by means of the distribution control files.


The distribution packages synch script (distsync) checks/synchronizes a local distribution mirror with a remote distribution mirror. It reads the distribution control files from the distributionlist file and checks if all packages listed are correctly present (exists, correct size and optionalMD5sum correct). Any wrong package is logged to STDERR. If a remote distribution mirror is specifiedany wrong package will be synchronized. If an outfile is given the name of the of any wrong package is written to this file so a synchronization could be made with a separate rsync command. Processing of the control files can be limited based on either dist/comp/arch. The packages within these control files can be limited based on their names or other informaotions.

When synchronizing first will be tried to move previous versions of the packages in place so bandwide could be saved (requires gzip --rsyncable). Of course previous version could either be removed or kept. Synchronizing for building different local mirrors also symlinks/hardlinks could be used.

Informations about packages could be changed through an exchange file. This file has the same structure the corresponding control file striped down to the package name and the to change informations of the packages. I.e. to change the section of a package this file has to contain the following package paragraph:

   Package: <packagename>
   Section: <change>

The structure of the local mirror could be transformed from a Debian mirror into a deviate structure. While no transformation produces the same structure, transform=dist produces the old dist structure (subdirectories according their section names) and transform=pool produces a complete pool structure (all old fashioned packages were moved into pool). The transform=user allows for a user definable transformation through the packagepath/packagename parameters in the config file (see below).

Besides using the current names for the control files (Packages, Sources) the script also processes a kind of names to allow for having multiple files in the same directory. These names are currently only used for the transformed mirrors and need to have the following naming:

   Packages_<dist>_<comp>_<arch> i.e. "Packages_woody_main_i386"
   Sources_<dist>_<comp>_source  i.e. "Sources_woody_main_source"
   Packages_<dist>_all_<arch>    i.e. "Packages_woody_all_i386"
   Packages_all_all_<arch>       i.e. "Packages_all_all_i386"
   Sources_all_all_source        i.e. "Sources_all_all_source"
Note: These naming schema is only necessary to link them to the corresponding names of an official Debian mirror.

If a distribution list file (i.e. "/etc/distributions.list") is defined, the control files where taken from this file. Besides using the legacy names or the above proposed names, any name could be choosen. For more informations see "distributions.list.txt".

The script does also have some special functions:
1. It can set the time (to 1. Jan. 1970) of any incorrect file.
2. It can set the user (i.e. "debadmin") of any correct file.

All the used values (like distnames, rsync options, etc.) are now configurable through a config file.


Written by Otto Wyss, mailto:wyo@users.sourceforge.net


 distsync		[-ACDfhkLlMPqStUVv] [-c=file] [-i=keywords] [-I=file] 
			[-m=url] [-o=file] [-T=keyword] [-u=user] [-x=file] 

 Checks/synchronizes a local distribution mirror with a remote distribution mirror. 
 The synchronization is driven by the distribution control files which where listed 
 in the distribution list file.

 -A|--actual-pkgs	Synchronize only actual installed packages
 -c|--config-file=file	Get the configuration from this file
 -C|--cleanup		Remove obsolete packages after synchronizing
 -D|--debug level	Debug functionality (level 1..9)
 -f|--force		Forces cleanup for the full local mirror
 -h|--help		Prints the usage
 -i|--include=keyword	Include any matching dist/comp/arch (',' separated)
 -I|--include-from=file	Include any packages (names, infos) listed in file
 -k|--keep		Keep old version of any package
 -l|--symlink		Make symlink instead of synchronizing
 -L|--hardlink		Make hardlink instead of synchronizing
 -m|--mirror=url	URL of a Debian mirror, pointing to base directory
 -M|--md5sum		Checks the MD5sum of each package
 -o|--output-to=file	Write wrong packages to file instead of synchronize
 -P|--packages		Process Debian packages (binaries)
 -q|--quiet		Don't show messages (decreases the verbose mode by one)
 -S|--sources		Process Debian sources
 -t|--settime		Set time to '1. Jan. 1970' if package not correct
 -T|--transform=keyword	Kind of tranformation (dist|pool|user)
 -u|--user=uid		Set user to new name (i.e. debadmin)
 -U|--utilities		Process Debian utilities (i.e. disk images)
 -v[v]			Verbose mode (-vv/-vvv more verbose)
 -V|--version		Version
 -x|--exchange-from=file Exchange control informations contained in file

 The syntax for the URL is either a local directory or an rsync location (with '::'). 
 Excluded packages cannot again be included (exclude has precedence).


Initial	2001-12-26	This is the first release of dpartialmirror

1.3.4	2002-01-03	Changed minor internals.

1.4.0	2002-01-08	All synchronizing scripts (debiansynch, debiandist and 
                        debianpool) combined into one script since most of the 
                        code was the same and all had now the same interface. 
                        Debiancompact change so it's able to compact any of the 
                        distribution structures and removed obsolete options.

1.4.1	2002-01-09	Some code fixes

1.4.2	2002-01-10	Fixed non-US, streamlined code
			Added config file

1.5.0	2002-01-11	Merged cleanup function into debiansynch.

1.5.1	2002-01-12	Fixed duplicate old versions. Code cleanup

1.5.2	2002-01-13	Fixed cleanup when transform where used.

1.5.3	2002-01-15	Removed left over "exit" statement
			Old control file not written over during synchronizing
			Packed control files are now packed again
			Code cleanup

1.5.4	2002-01-17	Fixed exclusion/inclusion
			Introduced -q (quiet) option
			Prepare for next major release

1.6.0	2002-01-18	Distributions list file integrated!

1.7.0	2002-01-20	Exchanged -e/-E and -i/-I options!

1.7.1	2002-01-21	Changed parameters dist... with deb_... in conf-file

1.7.2	2002-01-24	Sources were handled correct, but since there are only 
                        Sources.gz the new notation has to be used in 

1.7.3	2002-01-26	Fixed forced cleanup
			Prevented a direct force cleanup

1.7.4	2002-01-27	Includes preliminary man pages and an install script
			"sources.list" can be used instead of "distributions.list"

1.8.0	2002-01-31	Overhauled the conf-file, changed to Config::Inifile
			New feature: Synchronizes all installed items

1.8.1	2002-02-05	Fixed transform=user
			Declared release candidate

1.8.2	2002-02-09	Fixed bug in cleanup, no wrong removements anymore

1.8.3	2002-03-24	Lifted the restriction to use the -o option for a forced 
                        cleanup. Corrected the examples to acknowledge the change 
                        of the options -I and -E. Installs now into /usr/local/bin 
                        instead of /usr/bin.

1.8.4	2002-04-01	Exchange did not work for text including whitespace.
			All examples should now work.
			Small typo fix in usage text.

1.8.5	2002-04-03	Option -A doesn't reduce Packages file anymore.
			The package is renamed to "debiansynch" like the script.
			The location of the config file has changed to 

1.8.6	2002-04-25	Unlinks old control files only if the exists
			Option -U disabled until Debian provides utility control 

2.0.0	2002-05-15	Changed name to "distsync"
			Changed several wordings (hopefully better english)
			Removed scan for control files, they have to be declared
			Option -f only forces cleanup of full mirror
			Changed several config parameter names to better ones
			Config parameter defaults to more common values