migration-tools.git
2 years agoadded a find actor stat cat function, errors when it doesn't find since I use on... actor_stat_cat_functions
Rogan Hamby [Wed, 26 Apr 2017 19:50:48 +0000]
added a find actor stat cat function, errors when it doesn't find since I use on error stop

2 years agoquasi experimental stat cat and entry functions
Rogan Hamby [Wed, 26 Apr 2017 18:31:09 +0000]
quasi experimental stat cat and entry functions

2 years agoASSERT not supported until PG 9.5
Jason Etheridge [Tue, 11 Apr 2017 20:04:25 +0000]
ASSERT not supported until PG 9.5

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

2 years agosome useful wrappers for loading bre's
Jason Etheridge [Fri, 7 Apr 2017 14:53:13 +0000]
some useful wrappers for loading bre's

2 years ago"mig" tool
Jason Etheridge [Fri, 7 Apr 2017 12:31:31 +0000]
"mig" tool

Wraps some new and existing tools in a git-like command system,
currently great for quickly getting files into staging tables,
has aspirations for one day tracking mapping relationships and
doing code generation for mapping.

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

2 years agothird variant of migration_tools.assert with success message as 3rd argument
Jason Etheridge [Thu, 30 Mar 2017 17:32:44 +0000]
third variant of migration_tools.assert with success message as 3rd argument

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

2 years ago--force-utf8 in leader for marc_cleanup
Jason Etheridge [Thu, 30 Mar 2017 17:19:08 +0000]
--force-utf8 in leader for marc_cleanup

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

2 years agoassert tool for use with \set ON_ERROR_STOP on
Jason Etheridge [Thu, 30 Mar 2017 17:18:25 +0000]
assert tool for use with \set ON_ERROR_STOP on

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

2 years agouse UNLOGGED tables for the base staging tables
Jason Etheridge [Tue, 14 Mar 2017 17:22:51 +0000]
use UNLOGGED tables for the base staging tables

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

2 years agoadd note to readme
Rogan Hamby [Mon, 27 Feb 2017 13:38:29 +0000]
add note to readme

2 years agoadd vacuum analyze script for database maintenance in remove_ou scripts
Rogan Hamby [Mon, 27 Feb 2017 13:37:56 +0000]
add vacuum analyze script for database maintenance in remove_ou scripts

2 years agoadd a script to aid in generating the delete file for patrons in remove_ou scripts
Rogan Hamby [Mon, 27 Feb 2017 13:37:02 +0000]
add a script to aid in generating the delete file for patrons in remove_ou scripts

2 years agoadd disable/renable rule so that copy locations can be deleted in remove_ou scripts
Rogan Hamby [Mon, 27 Feb 2017 13:35:55 +0000]
add disable/renable rule so that copy locations can be deleted in remove_ou scripts

2 years agoadd removing usr circ history to remove_ou scripts
Rogan Hamby [Mon, 27 Feb 2017 13:34:45 +0000]
add removing usr circ history to remove_ou scripts

2 years agoadd removing call number suffixes and prefixes to remove_ou scripts
Rogan Hamby [Mon, 27 Feb 2017 13:33:59 +0000]
add removing call number suffixes and prefixes to remove_ou scripts

2 years agoremoved unncessary schema declared on column and fixed issue where value needed was...
Rogan Hamby [Thu, 9 Feb 2017 16:36:13 +0000]
removed unncessary schema declared on column and fixed issue where value needed was ID and org shortname was given

2 years agoremoved moving call numbers where items and call numbers were on incompatible orgs...
Rogan Hamby [Mon, 17 Oct 2016 18:50:31 +0000]
removed moving call numbers where items and call numbers were on incompatible orgs to be deleted

2 years agoadded check to make sure holds were fulfilled when deleting by fulfillment library
Rogan Hamby [Mon, 17 Oct 2016 18:47:19 +0000]
added check to make sure holds were fulfilled when deleting by fulfillment library

2 years agoremoved nulling of parent circs where child circ is at non-deleted library
Rogan Hamby [Mon, 17 Oct 2016 18:44:56 +0000]
removed nulling of parent circs where child circ is at non-deleted library

2 years agoRemove stat cats owned by to be deleted org unit, stat cat entries and entry user...
Rogan Hamby [Mon, 17 Oct 2016 16:25:44 +0000]
Remove stat cats owned by to be deleted org unit, stat cat entries and entry user mappings by those stat cats or patrons of those org units.

2 years agomove call numbers from to be deleted org to undeleted org when the copy is at an...
Rogan Hamby [Mon, 17 Oct 2016 16:03:20 +0000]
move call numbers from to be deleted org to undeleted org when the copy is at an undeleted org

2 years agoadd deleting holds by request lib
Rogan Hamby [Mon, 17 Oct 2016 15:03:02 +0000]
add deleting holds by request lib

2 years agoremove auditor history edits
Rogan Hamby [Mon, 17 Oct 2016 14:53:07 +0000]
remove auditor history edits

2 years agoremove holds based on fulfillment library
Rogan Hamby [Mon, 17 Oct 2016 14:34:18 +0000]
remove holds based on fulfillment library

2 years agoupdate the noncat circ code to note delete non cat circs when the staff member is...
Rogan Hamby [Mon, 17 Oct 2016 14:17:49 +0000]
update the noncat circ code to note delete non cat circs when the staff member is registered at the wrong library

2 years agodeletions of non cat circs
Rogan Hamby [Mon, 17 Oct 2016 14:08:39 +0000]
deletions of non cat circs

2 years agoremove albermale references by putting in a variable and deleting usr messages
Rogan Hamby [Mon, 17 Oct 2016 14:03:24 +0000]
remove albermale references by putting in a variable and deleting usr messages

2 years agoadded line to handle child checkouts where parent checkout is at the org being delete...
Rogan Hamby [Mon, 17 Oct 2016 12:48:24 +0000]
added line to handle child checkouts where parent checkout is at the org being deleted but child is not by nulling parent circ, unfortunately no way to indicate this

2 years agoOU data removal script
Galen Charlton [Fri, 14 Oct 2016 15:17:15 +0000]
OU data removal script

This is a set of SQL scripts to be run, in order, to completely
remove data belonging to an OU and its descendants in an Evergreen
database.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

2 years agofilter out linefeeds and carriage returns from holding tag values
Jason Etheridge [Wed, 14 Sep 2016 13:59:42 +0000]
filter out linefeeds and carriage returns from holding tag values

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

2 years agofilter out tabs from holding tag values
Jason Etheridge [Wed, 14 Sep 2016 13:33:54 +0000]
filter out tabs from holding tag values

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

2 years agominor whitespace correction
Galen Charlton [Mon, 8 Aug 2016 18:25:17 +0000]
minor whitespace correction

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

2 years agoMake --wait optional in eg_staged_bib_overlay.
Jason Stephenson [Sat, 6 Aug 2016 21:28:47 +0000]
Make --wait optional in eg_staged_bib_overlay.

This commit only causes load bibs to wait if the --wait option is
specified with a non-zero value.  It also changes the default for the
global $wait variable that the option is read into to zero.

Signed-off-by: Jason Stephenson <jason@sigio.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>

2 years agoPerl eval improvements in eg_staged_bib_overlay.
Jason Stephenson [Sun, 7 Aug 2016 14:23:42 +0000]
Perl eval improvements in eg_staged_bib_overlay.

Two changes to improve the eval conditions added in the previous
commit:

1. Both error handlers now warn with the same message:

   Record $i is bad: $@; skipping.

Thereby reporting the original message in case the user wants to try
and recover the records later.

2. If a bib record's 901$c is either missing or not numeric, die() is
called with a message to that effect.  This eliminates the Perl
warnings about exiting an eval block with next.

Signed-off-by: Jason Stephenson <jason@sigio.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>

2 years agoUse eval in eg_staged_bib_overlay.
Jason Stephenson [Sat, 6 Aug 2016 21:10:12 +0000]
Use eval in eg_staged_bib_overlay.

Wrap the parsing of MARC from input files in eval blocks with an error
handler to skip to the next record.  I have encounterd bad records
that have stopped the processing of the rest of the file.  This
occurred the very first time I used this script.

Signed-off-by: Jason Stephenson <jason@sigio.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>

2 years agoconvert DestinyCustomReport XML to TSV
Jason Etheridge [Tue, 19 Jul 2016 20:17:49 +0000]
convert DestinyCustomReport XML to TSV

3 years agoadd --fix-subfield
Jason Etheridge [Mon, 6 Jun 2016 19:59:44 +0000]
add --fix-subfield

and tweak some comments and command-line help

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agoexample Follet/Destiny 852 map
Jason Etheridge [Fri, 27 May 2016 19:17:59 +0000]
example Follet/Destiny 852 map

3 years agosyntax in INHERIT example
Jason Etheridge [Fri, 27 May 2016 19:17:32 +0000]
syntax in INHERIT example

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agoeg_staged_bib_overlay: add --link-skipped
Galen Charlton [Wed, 18 May 2016 16:04:23 +0000]
eg_staged_bib_overlay: add --link-skipped

The new --link-skipped switch modifies the link_auth_bibs
action to say that bibs that were skipped due to having been
edited after the cutoff should be linked to authorities (rather
than linking imported bibs).

The use case for this: the library decides that they don't
want to scribble over the skipped bibs, but still want to
take advantage of any new authorities that correspond to them.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

3 years agoeg_staged_bib_overlay: add actions to export skipped records
Galen Charlton [Mon, 16 May 2016 17:31:43 +0000]
eg_staged_bib_overlay: add actions to export skipped records

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

3 years agoeg_staged_bib_overlay: fix help
Galen Charlton [Mon, 2 May 2016 18:57:35 +0000]
eg_staged_bib_overlay: fix help

Corrected the description of the --overlay_auths_stage1
and --overlay_auths_stage2 switches.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

3 years agoeg_staged_bib_overlay: add link_auth_bib action
Galen Charlton [Mon, 18 Apr 2016 19:57:17 +0000]
eg_staged_bib_overlay: add link_auth_bib action

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

3 years agostaged record loader: add various authority record actions
Galen Charlton [Mon, 11 Apr 2016 19:32:52 +0000]
staged record loader: add various authority record actions

--action stage_auths          - load MARC authorities into staging
                                table
--action match_auths          - identify matches with authority
                                records already present in the
                                Evergreen database; matching is
                                based on LCCN, cancelled LCCN, and
                                main heading.
--action load_new_auths       - load new (unmatched) authorities
--action overlay_auths_stage1 - overlay based on LCCN where
                                heading has not change; this step
                                disables propagation to bib records
--action overlay_auths_stage2 - overlay based on LCCN where heading
                                has NOT changed; propagates changes
                                to bib records
--action overlay_auths_stage3 - overlay for records where a cancelled
                                LCCN is replaced with a new one
--action link_auth_auth       - run authority_authority_linker.pl for
                                the authorities that were overlaid
                                or added in this batch.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

3 years agotweaks to name parser
Jason Etheridge [Mon, 11 Apr 2016 15:53:33 +0000]
tweaks to name parser

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agostaged bib overlay: tweak bib filtering
Galen Charlton [Mon, 11 Apr 2016 14:18:18 +0000]
staged bib overlay: tweak bib filtering

Filtering now always ignores staged bibs that are
imported or already marked as not to be imported.

Also, bibs whose XML is not well-formed are now
excluded.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

3 years agoupdate usage information
Galen Charlton [Tue, 5 Apr 2016 14:26:22 +0000]
update usage information

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

3 years agonew utility to overlay bib records in an Evergreen database
Galen Charlton [Tue, 5 Apr 2016 14:24:47 +0000]
new utility to overlay bib records in an Evergreen database

Utility to stage and overlay bib records in an Evergreen database. This
expects that the incoming records will have been previously exported
from that Evergreen database and modified in some fashion (e.g., for
authority record processing) and that the bib ID can be found in the
901$c subfield.

This program has several modes controlled by the --action switch:

  --action stage_bibs  - load MARC bib records into a staging table
  --action filter_bibs - mark previously staged bibs that should
                         be excluded from a subsequent load, either
                         because the target bib is deleted in Evergreen
                         or the record was modified after a date
                         specified by the --cutoff switch
  --action load_bibs   - overlay bib records using a previously staged
                         batch, one at a time. After each bib, it will
                         wait the number of seconds specified by the
                         --wait switch.

Several switches are used regardless of the specified action:

  --schema  - Pg schema in which staging table will live; should be
              created beforehand
  --batch   - name of bib batch; will also be used as the name
              of the staging table
  --db      - database name
  --dbuser  - database user
  --dbpw    - database password
  --dbhost  - database host

Examples:

./eg_staged_bib_overlay --schema bib_load --batch bibs_2016_01 --db evergreen \
   --dbuser evergreen --dbpw evergreen --dbhost localhost \
   --action stage_bibs -- file1.mrc file2.mrc [...]

./eg_staged_bib_overlay --schema bib_load --batch bibs_2016_01 --db evergreen \
   --dbuser evergreen --dbpw evergreen --dbhost localhost \
   --action filter_bibs --cutoff 2016-01-02

./eg_staged_bib_overlay --schema bib_load --batch bibs_2016_01 --db evergreen \
   --dbuser evergreen --dbpw evergreen --dbhost localhost \
   --action load_bibs

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

3 years agoteach export_evergreen_library how to emit serials data
Galen Charlton [Thu, 17 Mar 2016 16:59:53 +0000]
teach export_evergreen_library how to emit serials data

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

3 years agoacq.fund_tag_map
Jason Etheridge [Fri, 11 Mar 2016 21:31:11 +0000]
acq.fund_tag_map

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agoadd acq.provider_contact and acq.provider_contact_address to base staging table
Jason Etheridge [Fri, 11 Mar 2016 19:22:37 +0000]
add acq.provider_contact and acq.provider_contact_address to base staging table

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agoadd action.transit_copy and action.hold_transit_copy to base tables
Jason Etheridge [Mon, 29 Feb 2016 14:46:47 +0000]
add action.transit_copy and action.hold_transit_copy to base tables

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agoadd action.hold_request_note to the list of base tables
Jason Etheridge [Mon, 29 Feb 2016 14:39:56 +0000]
add action.hold_request_note to the list of base tables

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agomigration_tools.attempt_hz_date
Ben Ostrowsky [Fri, 19 Feb 2016 02:33:10 +0000]
migration_tools.attempt_hz_date

3 years agotweak for case when just last name
Jason Etheridge [Fri, 19 Feb 2016 02:07:04 +0000]
tweak for case when just last name

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agomigration_tools.name_parse_out_last_first_middle_and_random_affix for LAST FIRST...
Jason Etheridge [Fri, 19 Feb 2016 02:01:09 +0000]
migration_tools.name_parse_out_last_first_middle_and_random_affix for LAST FIRST MIDDLE with no comma

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agomigration_tools.find_hold_matrix_matchpoint2
Jason Etheridge [Thu, 28 Jan 2016 17:08:58 +0000]
migration_tools.find_hold_matrix_matchpoint2

another convenience function

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agoremove config fig.circ_matrix_circ_mod_test
Jason Etheridge [Thu, 24 Sep 2015 04:45:37 +0000]
remove config fig.circ_matrix_circ_mod_test

and _map from the list of base tables so that .build() will work with newer
versions of EG

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agomore money converters
Jason Etheridge [Mon, 10 Aug 2015 03:45:03 +0000]
more money converters

migration_tools.attempt_money_from_pennies
and migration_tools.attempt_money_from_pennies6

Avoids dividing by 100 when doing the conversion

2nd argument, the fail value, should not be pennies

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

3 years agoduplicate_template_but_change_delay
Jason Etheridge [Thu, 6 Aug 2015 03:22:21 +0000]
duplicate_template_but_change_delay

Let's us get around duplicate key issues in some cases by tweaking the delay as
part of the cloning process.

Example:

    ubuntudeskfx=# SELECT * FROM migration_tools.duplicate_template(5,'{3,4}','00:30:00'::INTERVAL);

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoMinor typo fix in export_evergreen_library
Mike Rylander [Thu, 18 Jun 2015 13:49:27 +0000]
Minor typo fix in export_evergreen_library

Signed-off-by: Mike Rylander <mrylander@gmail.com>

4 years agotweak migration_tools.insert_856_9
Jason Etheridge [Mon, 20 Apr 2015 16:10:02 +0000]
tweak migration_tools.insert_856_9

rename original to migration_tools.insert_856_9_conditional, since it has
behavior based on 856$u content.  New version simply adds a $9 if one does
not already exist (regardless of org unit referenced)

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agohandle dates like 051980
Jason Etheridge [Tue, 24 Mar 2015 20:17:16 +0000]
handle dates like 051980

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoadd some in-db matchpoint tables, for potential ease of staging policy rules
Jason Etheridge [Wed, 18 Mar 2015 14:49:20 +0000]
add some in-db matchpoint tables, for potential ease of staging policy rules

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoconvenience wrappers for finding in-db rules
Jason Etheridge [Mon, 16 Mar 2015 18:22:57 +0000]
convenience wrappers for finding in-db rules

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agomake migration_tools.apply_circ_matrix_to_specific_circ use the more recent apply_cir...
Jason Etheridge [Mon, 16 Mar 2015 17:56:21 +0000]
make migration_tools.apply_circ_matrix_to_specific_circ use the more recent apply_circ_matrix code

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoadd a delimiter paramater, and a variant that returns an array of tags
Jason Etheridge [Wed, 25 Feb 2015 15:54:20 +0000]
add a delimiter paramater, and a variant that returns an array of tags

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years ago.get_marc_tag
Jason Etheridge [Sat, 21 Feb 2015 01:53:35 +0000]
.get_marc_tag

Useful since you can't get unnormalized values out of metabib.full_rec

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoMerge branch 'master' of git.esilibrary.com:migration-tools
Jason Etheridge [Thu, 5 Feb 2015 17:30:30 +0000]
Merge branch 'master' of git.esilibrary.com:migration-tools

4 years agomigration_tools.apply_circ_matrix_to_specific_circ
Jason Etheridge [Thu, 5 Feb 2015 17:30:26 +0000]
migration_tools.apply_circ_matrix_to_specific_circ

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoproc to clone a/t event definitions
Jason Etheridge [Tue, 3 Feb 2015 17:13:21 +0000]
proc to clone a/t event definitions

ubuntudeskfx=# SELECT * FROM migration_tools.duplicate_template(5,'{3,4}');
INFO:  created atevdef with id = 112
INFO:  created atevdef with id = 113
INFO:  -- UPDATE action_trigger.event_definition SET active = CASE WHEN id = 5 THEN FALSE ELSE TRUE END WHERE id in (5,112,113);
 duplicate_template
--------------------

(1 row)

ubuntudeskfx=# UPDATE action_trigger.event_definition SET active = CASE WHEN id = 5 THEN FALSE ELSE TRUE END WHERE id in (5,112,113);
UPDATE 3

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoexample script showing how to use 'fingerprinter' to deduplicate bib records
Galen Charlton [Thu, 29 Jan 2015 22:51:08 +0000]
example script showing how to use 'fingerprinter' to deduplicate bib records

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

4 years agolet's strip commas as well with attempt_money
Jason Etheridge [Wed, 14 Jan 2015 21:58:44 +0000]
let's strip commas as well with attempt_money

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoan attempt_money method for a smaller numeric, given what we use for billings and...
Jason Etheridge [Wed, 14 Jan 2015 21:53:55 +0000]
an attempt_money method for a smaller numeric, given what we use for billings and payments

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agolet migration_tools.attempt_money strip dollar signs for us
Jason Etheridge [Wed, 14 Jan 2015 21:51:42 +0000]
let migration_tools.attempt_money strip dollar signs for us

4 years agoinclude affixes with _callnum_label_once_per_lib index
Jason Etheridge [Thu, 20 Nov 2014 22:16:30 +0000]
include affixes with _callnum_label_once_per_lib index

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoadd tables for call number affixes
Jason Etheridge [Thu, 20 Nov 2014 13:52:08 +0000]
add tables for call number affixes

4 years agoadd routine to calculate Code 39 mod 43 barcode checkdigits
Galen Charlton [Mon, 22 Sep 2014 18:44:03 +0000]
add routine to calculate Code 39 mod 43 barcode checkdigits

migration_tools.add_code39mod43_checkdigit($barcode);
If the source string is 13 or 14 characters long and contains only valid
Code 39 mod 43 characters, adds or replaces the 14th
character with a checkdigit computed according to the usual algorithm for library barcodes
using the Code 39 mod 43 symbology - see <http://en.wikipedia.org/wiki/Code_39#Code_39_mod_43>.  If the
input string does not meet those requirements, it is returned unchanged.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

4 years agoSimple script to export records belonging to a specific Evergreen OU and its descendents.
Galen Charlton [Thu, 28 Aug 2014 21:08:09 +0000]
Simple script to export records belonging to a specific Evergreen OU and its descendents.

Usage: export_evergreen_library shortname scratch_dir output_dir db_conn_params

For example:

$ mkdir inter out
$ ./export_evergreen_library SYS1 inter out '-U evergreen'

The script emits MARC bibs, undeleted patrons, all items for the OU, all loans
for the OU's patrons and items, current hold requests, and patron billings.

The script expects to be run on a box that can access the Evergreen database
via psql and an OpenSRF stack.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

4 years agofix construct that doesn't work in Perl 5.10
Galen Charlton [Fri, 1 Aug 2014 19:22:04 +0000]
fix construct that doesn't work in Perl 5.10

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

4 years agoimprove tool for cleaning up merge maps
Galen Charlton [Wed, 30 Jul 2014 20:53:47 +0000]
improve tool for cleaning up merge maps

The map cleaner/merger now correctly handles
cases where a given subordinate bib has a lead
bib that itself has a different lead bib.  It
also excludes cyclical loops in the merge map.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

4 years agomake the fingerprinter normalize OCLC control numbers
Galen Charlton [Wed, 30 Jul 2014 19:53:02 +0000]
make the fingerprinter normalize OCLC control numbers

Various ways of representing the same control number
are now normalized to "(OCoLC)" followed by the numeric
portion of the control number.

This patch also introduces a new module, Equinox::Migration::Utils,
which is meant for miscellaneous utility routines.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agoenhancements to patron/circ purge
Galen Charlton [Thu, 2 May 2013 16:13:17 +0000]
enhancements to patron/circ purge

- add a couple notes and instructions
- add a couple more tables to clear out
- use TRUNCATE where possible

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agoimprovements to bib nuking script
Galen Charlton [Thu, 2 May 2013 15:43:03 +0000]
improvements to bib nuking script

- truncate a few tables that don't have explicit FK
  links to bibs or copies
- nuke authority records
- inform the sysadmin of the next steps to take to
  complete the purge

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agodump_oracle_table_for_pg: add support for MARC8
Galen Charlton [Wed, 17 Apr 2013 00:52:33 +0000]
dump_oracle_table_for_pg: add support for MARC8

The dumper now groks --source-charset marc8

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agoNew tool: dump_oracle_query_output
Galen Charlton [Tue, 16 Apr 2013 22:42:30 +0000]
New tool: dump_oracle_query_output

Dumps results of query on Oracle database to file for loading into PostgreSQL

Usage: ./dump_oracle_query_output \
    [--sid oracle_sid] [--host oracle_host] --user oracle_user --pw oracle_password \
    --query sql_query \
    --out output_tsv_file [--help]

This exists because sqlplus lacks an easy way to dump arbitrary
query output to TSV or CSV.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agodump_oracle_table_for_pg: try to ensure dump output is UTF-8
Galen Charlton [Tue, 16 Apr 2013 21:02:25 +0000]
dump_oracle_table_for_pg: try to ensure dump output is UTF-8

Adds a --source-charset command-line parameter to specify
the source character encoding.  Note that CLOB/BLOB columns
are not considered (not that they were before this commit).

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agodump_oracle_table_for_pg: recognize CHAR columns
Galen Charlton [Tue, 16 Apr 2013 20:40:03 +0000]
dump_oracle_table_for_pg: recognize CHAR columns

Also don't complain if integer columns don't specify
a scale.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agowhen parsing state from city, anchor pattern on right
Galen Charlton [Fri, 8 Mar 2013 18:20:47 +0000]
when parsing state from city, anchor pattern on right

This avoids "ANCHOR POINT AK" from becoming {"ANCHORINT", "AK"}

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agomap Oracle NUMBER columns to correct Pg type
Galen Charlton [Wed, 6 Mar 2013 18:38:01 +0000]
map Oracle NUMBER columns to correct Pg type

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agonew utility: dump_oracle_table_for_pg
Galen Charlton [Wed, 6 Mar 2013 18:19:07 +0000]
new utility: dump_oracle_table_for_pg

This utility dumps the contents of an Oracle table
to file for loading into PostgreSQL.

Usage: dump_oracle_table_for_pg \
    [--sid oracle_sid] [--host oracle_host] --user oracle_user --pw oracle_password \
    --table oracle_table_name \
    --pg-table destination_pg_table_name \
    --out output_tsv_file --sql output_table_create_sql_file \
    [--column-prefix column_prefix] [--inherits-from base_pg_table] [--help]

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agorework attempt_cast function
Galen Charlton [Wed, 26 Dec 2012 17:00:32 +0000]
rework attempt_cast function

This function now returns the original value if it can
be cast to the desired type, and NULL otherwise.  For example:

UPDATE foo
SET price = migration_tools.attempt_cast(l_price, 'NUMERIC(8,2)')::NUMERIC(8,2);

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agoimprove marc_parses() test
Galen Charlton [Fri, 19 Oct 2012 14:10:29 +0000]
improve marc_parses() test

If the Leader/09 is not 'a' and the record has diacritics,
MARC::Record and friends don't complain until you try to
emit a MARCXML record.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agostart work on a function for creating staff users in bulk
Galen Charlton [Wed, 3 Oct 2012 20:24:52 +0000]
start work on a function for creating staff users in bulk

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agonew function to set indicators in MARC fields
Galen Charlton [Mon, 10 Sep 2012 15:42:20 +0000]
new function to set indicators in MARC fields

migration_tools.set_leader(marc, tag, indicator_pos, value):

Set indicator value of a specified MARC field.
The first argument is a MARCXML string.
The second argument is a MARC tag.
The third argument is the indicator position, either 1 or 2.
The fourth argument is the character to set the indicator value to.
All occurences of the specified field will be changed.
The function returns the revised MARCXML string.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agocreate base staging tables for acquisitions
Galen Charlton [Thu, 30 Aug 2012 20:13:31 +0000]
create base staging tables for acquisitions

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agotypo fix
Galen Charlton [Wed, 29 Aug 2012 21:16:39 +0000]
typo fix

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agoteach migration_tools.change_call_number() about label classes
Galen Charlton [Wed, 29 Aug 2012 21:01:54 +0000]
teach migration_tools.change_call_number() about label classes

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

6 years agoMerge branch 'master' of git.esilibrary.com:migration-tools
Galen Charlton [Fri, 17 Aug 2012 20:11:34 +0000]
Merge branch 'master' of git.esilibrary.com:migration-tools

6 years agoremove last trace of vestigial bib modifier
Galen Charlton [Fri, 17 Aug 2012 20:10:25 +0000]
remove last trace of vestigial bib modifier

This modifier was removed in commit 8e63d0bdd; this commit
just removes the last reference.  Thanks to Justin Hopkins
for spotting this.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>