Planning to bulk import your patrons? Make sure you do not have in-line line breaks in your data.

In-line line breaks in a CSV file can really send your Koha patron import script into a tailspin. Here is what you need to watch out for and the couple of other gotchas which will make you upgrade your Koha instance if the version you are using is less than 3.22.7.

Last week a friend working at a local college approached me for a spot of help. He was trying to import his patrons into Koha but was failing miserably. After he nearly got his head snapped off (Me: Do I look like I’m in the fortune telling profession???) he agreed to send over his data – an MS-Excel sheet for me to take a look at.

I pulled up a 3.22.6 instance I had laying around and tried to import his data. Quite expectedly, there were errors galore and the pretty much the same ones he was complaining about.

blog_patron_1

Hang on! categorycode, branchcode and surname fields were NOT missing in *any* single record. So what was going on here??? The most interesting to note here is that patron importer script said :

272 not imported because they are not in the expected format

272 records parsed

Now this was really something as the total number of student records in that patron uploader CSV file were only 144. So where does the number 272 come from?

The answer to this was easy to find. My friend’s data had several records in a rather bad shape – they had embedded line-breaks within the cell. I’ve highlighted the first few of the badly formatted cells with yellow in the screenshot below.

blog_patron_3A

So, I copied the first 28 records over to a new file, ran a hackish utility script to clean out the line breaks and saved these 28 records as a new file and proceeded to upload it. This time of course “the fat lady sang”[1] i.e. the records got imported nice and we were done!. 😀

blog_patron_3NOTE:  Of course while doing that we encountered a few Koha bugs as well – Bug id 15840 and Bug id 16426. The work-around mentioned in comment #16 of the latter bug, by Koha QA Manager Katrin Fischer holds good, in case you get stuck here and can’t immediately upgrade. Otherwise to avoid these to bugs, your real option is to upgrade your Koha instance, something that I’m going to recommend to my friend (aside from him fixing his data).

Reference: [1] Wikipedia “It ain’t over till the fat lady sings”

 

Easy peasy way of automating remote backup on Google Drive for your Koha database

This post discusses how to automate your Koha ILS’s MySQL database backup on to Google drive and send an email when it is complete. It shows how you can take advantage of Google Drive’s 15GB space for free (Dropbox only gives you 2GB on the free access) and do it all from the command line and save the much needed RAM for your Koha server rather than waste it on the GUI, which is also a security risk. Further this attempts to introduce the novice readers into details of the commands they are supposed to follow, with further reading resources, should they be inclined to learn more.

Having your Koha ILS database to be regularly backed up on to remote, cloud storage is an excellent idea. By doing so you ensure a critical off-site, disaster recovery measure, which is very good. However, as with all things human, if we leave it on ourselves to do it, there will come to pass a time when we will (a) forget to do it or (b) be unable to do it for some
reason. As we all know good ol’ Captain Murphy’s Law[1] will strike us whenever we are least prepared; in this case typically that one time we forgot or were unable to take the backup, the darned thing will crash!

So backup automation is key. Not only it ensures regularity without fail. It also removed one more essential chore from our immediate plate, thus leaving us free to do other things without feeling guilty over this key housekeeping chore.

Cloud backup – Google vs Dropbox

Dropbox and Google Drive comes across as immediate choice of cloud based backup. However, their free editions differ [2]… only by about 13GB of space between them. So for long-term online backup Google Drive is the de-facto choice.

Our objective

So, here is what we set out to do:

  1. create a datetime stamped backup of the database; (so we can tell just by seeing the filename when the backup was taken)
  2. compress it with bzip2 utility; (so all those loooooong lines of SQL text do not take up so much space, a text file can compress up to within 10% of it original size)
  3. upload it to a specified folder on Google Drive; (so that all our backups remain in one place, date-wise)
  4. email the user that the remote backup process is complete. (so when we outside or on vacation and don’t have access to our workstation, we still get a notification when it was completed and if we don’t get one, then that something certainly went wrong and someone should do something about it)

And of course, since we are talking about making this happen everyday at the same time, we need to create a cron job that will deliver all of 1, 2, 3 and 4 to us in a single neat little command.

As you all know, no self respecting system administrator will ever be caught running the X11 windowing system on a production server. So we are going to do these the way real system admins do: from the command line.

NOTE: X11 is the geekspeak for the Graphical User Interface (GUI) environment we see e.g. when we log into an Ubuntu Desktop (which is typically the Unity desktop)

Command line in this day and age? Are you nuts???

No! And here is the reason. X11 is not only an inherently insecure protocol that puts your production system at risk, it is also (compared with a command line only system) a tremendous resource hog! We all know that more free memory (RAM) is usually-a-good-thing ™, so instead of wasting our precious RAM on running a GUI (and all the unnecessary software along with it making it slow *and* insecure) we are going to show you how to do this all from a command line. One other thing: if you ever need the assistance of an expert, you will find that command line setups are also easier to debug (for an expert), after all, aren’t they always asking you to check your “logs”? All those are after all command line output. So like the Chloromint ad below, please don’t ask us again why we love the command line! 😉

Preparations

We want a normal user account with no admin privileges; say in our case we will call it l2c2backup and we will do it from the terminal using the adduser l2c2backup command. See below:

blog_01

Next up, we need to switch over to the new user account and create a synchronization folder for Google drive.

blog_02

At this point, we’ll press “Ctrl+D” and exit from the l2c2backup user and come back to the root user or sudo user, for we now need to install a command line google drive client on our system. We are going to use the (almost) official Google Drive command line client for Linux known simply as “drive” and available from https://github.com/odeke-em/drive

Since we are using Debian, we have the advantage of using the pre-built binaries, which we shall install in the following manner by executing in turn each of the commands:
# apt-get install software-properties-common
# apt-add-repository 'deb http://shaggytwodope.github.io/repo ./'
# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7086E9CC7EC3233B
# apt-key update
# apt-get update
# sudo apt-get install drive

NOTE:If you are using Ubuntu or other mainstream Linux distributions, you can use the instructions given here on the Platform Packages page.

Once we have completed installation of “drive“, we now need to go back to our /home/l2c2backup/gdrive folder as the user l2c2backup and initialize the sync folder (i.e. /home/l2c2backup/gdrive) using the command “drive init

blog_03

Copy the really long URL that the command tells you to visit and open it in your web browser. You will see an application authorization dialog screen come up, click on the “Allow” button.

blog_03A

NOTE: Before pasting the URL, you must make sure that at this point you are logged in into the actual Google user account where you want to send the backups to. Don’t make a mess here.

Assuming you did everything as I have mentioned so far, you will be automatically redirected to the page with the authorization key. It will look pretty much like the one below. Of course, every request will generate a separate access authorization key, so use the one generated specifically against your request.

blog_03B

Copy this key and paste it back at the prompt in your terminal window and press <ENTER>. DO NOT TRY TO TYPE IT OUT BY HAND, COPY-N-PASTE IS THE ONLY WAY HERE!

If you have done everything alright then you should be back at the command prompt without any error or any other message. Your sync folder should now be ready.

Putting our solution together

Now that we have the Google Drive sync ready, it is time to look at each piece of our basic requirement.

1. Creating a datetime stamped backup of our database

First we need to create the name of our output file for the MySQL backup. For this we shall use this: BACKFILE="<dbname>.$(date +"%Y%m%d_%H%M%S").sql;. The date format will give us a datetime string formatted as “20160723_000001” when the date & time is 12:00:01 (AM) on 23-July-2016. For this example, let us assume that the BACKFILE environment variable will hold the value: koha_ghci.20160723_000001.sql.

Note: replace <dbname> with the actual name of your Koha database, which in our case is koha_ghci. So, the syntax for us looked like: BACKFILE=koha_ghci.$(date +"%Y%m%d_%H%M%S").sql;. If you want to learn more about the format specific to the date command, you can read up this.

Next we will create the actual db backup using the datetime stamped output filename we just created. For that mysqldump -u<mysql_db_username> -p<mysql_db_passwd> <dbname> > /home/l2c2backup/gdrive/$BACKFILE.

Note: replace the <mysql_db_username>, <mysql_db_passwd> and <dbname> placeholders with your actual values. In our example case, the actual backup command string looked like this: mysqldump -ukoha_ghci -pASx2xvercbHXzs2dP koha_ghci > /home/l2c2backup/gdrive/$BACKFILE.

2. Compressing our SQL export

The previous step had exported our koha_ghci database as koha_ghci.20160723_000001.sql. We shall now compress this with bzip2 /home/l2c2backup/gdrive/$BACKFILE, which will give us the compressed file koha_ghci.20160723_000001.sql.bz2

3. Upload the compressed SQL backup to Google drive

Before we proceed with the actual upload, we should create a dedicated directory *on* our actual Google drive to store our backups. Lets call this directory as DBBACKUPS and create it on our online Google Drive space. It should be mentioned here that the command for upload using this library we are using, takes the form of drive push --destination <remote_folder_name> <full_path_to_compressed_file>. This code will ask for confirmation and we need to pass “Y” for yes before it will proceed. So we need to take care out that by adding echo Y | before the drive push command.

So in our case it will be echo Y | drive push --destination DBBACKUPS /home/l2c2backup/gdrive/$BACKFILE.bz2

Note:If you wish to learn about the other various options you can additionally use with drive push, I suggest you read this for the details.

4. Sending an email when the upload is done.

We are not running a dedicated, full fledged mail server like say Postfix on this box. Rather we are using the lightweight mstmp-mta with our Gmail account as the mail relay. If you want to know how to configure it, I suggest that you read this tutorial, ignoring the “mutt” part which you do not require. It is very simple. We had email sending working in under a minute. That’s just how long it took use to configure it.

Note: Just remember you *must* have openssl installed otherwise you will never be able to talk to GMail. And also you will need to go to your Google account and enable support for that Google likes to call “less secure apps” (which means any app that does use Google’s OAuth2 protocol for authentication. You will be authenticating over TLS and it is a perfectly safe thing to do, so just ignore Google’s ominous tone and enable “less secure apps”.

Now that we have msmtp-mta up and running, we will send out that email using this: printf "To: <recipient_email_address>\nFrom: <your_gmail_address>\nSubject: <dbname> db backed up on GDrive\n\nSee filename $BACKFILE.bz2 on DBBACKUPS folder on Google Drive of <your_gmail_address>.\n\nBackup synced at $(date +"%Y-%m-%d %H:%M:%S")" | msmtp <recipient_email_address>

In our case that happened to be printf "To: monitoring@l2c2.co.in\nFrom: indradg@gmail.com\nSubject: KOHA_GHCI db backed up on GDrive\n\nSee filename $BACKFILE.bz2 on DBBACKUPS folder on Google Drive of indradg@gmail.com.\n\nBackup synced at $(date +"%Y-%m-%d %H:%M:%S")" | msmtp indradg@l2c2.co.in.

5. Putting it all together

Now that we have all the parts of the puzzle in place, it is time to assemble it into a single piece. And the way, it worked for us was BACKFILE=koha_ghci.$(date +"%Y%m%d_%H%M%S").sql; mysqldump -ukoha_ghci -pASx2xvercbHXzs2dP koha_ghci > /home/l2c2backup/gdrive/$BACKFILE && bzip2 /home/l2c2backup/gdrive/$BACKFILE && echo Y | drive push --destination DBBACKUPS /home/l2c2backup/gdrive/$BACKFILE.bz2 && printf "To: indradg@l2c2.co.in\nFrom: indradg@gmail.com\nSubject: KOHA_GHCI db backed up on GDrive\n\nSee filename $BACKFILE.bz2 on DBBACKUPS folder on Google Drive of indradg@gmail.com.\n\nBackup synced at $(date +"%Y-%m-%d %H:%M:%S")" | msmtp indradg@l2c2.co.in

Note: The reason we used the “&&” is that in BASH it stands for what is called as “Logical AND”. In simple English this merely means that unless the previous command is not not executed successfully, whatever comes next simply won’t execute.

A BASH script and a cron job

We placed this one-liner script that cobbled together into the following BASH script which we named as “backuptogoogle.sh” and placed it in the folder /usr/local/bin after setting its execution bit on with chmod a+x /usr/local/bin/backuptogoogle.sh

#!/bin/bash
BACKFILE=koha_ghci.$(date +"%Y%m%d_%H%M%S").sql; mysqldump -ukoha_ghci -pASx2xvercbHXzs2dP koha_ghci > /home/l2c2backup/gdrive/$BACKFILE && bzip2 /home/l2c2backup/gdrive/$BACKFILE  && echo Y | drive push --destination DBBACKUPS /home/l2c2backup/gdrive/$BACKFILE.bz2 && printf "To: indradg@l2c2.co.in\nFrom: indradg@gmail.com\nSubject: KOHA_GHCI db backed up on GDrive\n\nSee filename $BACKFILE.bz2 on DBBACKUPS folder on Google Drive of indradg@gmail.com.\n\nBackup synced at $(date +"%Y-%m-%d %H:%M:%S")" | msmtp indradg@l2c2.co.in

We setup a root user cron job with crontab -e and adding the following line and saving it.

@daily /usr/local/bin/backuptogoogle.sh

Note: The @daily shortcut will execute our script exactly at mid-night everyday. If you want to know what are the other useful cronism shortcuts, I suggest you read this useful post by my Koha colleague and good friend D. Ruth Bavousett over here.

Backup automation from command line

If you have been able to follow the instructions by suitably modifying them to your specific settings, you have just achieved backup automation from the command line. Like I said… It’s Easy Peasy!!! 😀

References:

[1] https://en.wikipedia.org/wiki/Murphy%27s_law

[2] http://www.cloudwards.net/dropbox-vs-google-drive/#features

Adding the item type filter to OPAC masthead search

Earlier today my good friend Vimal Kumar Vazaphally posted a question here about how to add the the item type (mc-itype in koha search speak) filter as a dropdown to the default main masthead search in the Koha OPAC.

Fig. 1: The default OPAC masthead search bar.
Fig. 1: The default OPAC masthead search bar.

RTFM and RTFM often!!! It may save your life!

<rant>The discussion that pursued on the FB group made something very clear. People forget to read the fine manual and when they do, they do not read “between the lines”. Trust me on this one, the Koha user manual is truly a ginormous treasure trove, if you take the pains to read it.</rant>

Ok! here is why I said that people really need to RTFM. The solution to the problem which Vimal shared can easily be extrapolated from this section in the manual – Appendix P: Extending KohaNewest title pulldown (#KohaTrivia it is based on a July 2012 blog post by Nicole C. Engard (Koha’s indefatigable Documentation manager).

“Reading between the lines” of a given solution

Here was the catch: Nicole’s solution pulled out only the newest arrivals of each itemtype, whereas we need it to pull out everything tagged to an itemtype, whether new or old. The second catch was that in Nicole’s example, she added plain HTML to the OpacNav system preference. However we are going to add it via the jquery / javascript route using the opacuserjs system preference.

We will follow Nicole’s example and pull out all item types that we need to populate the drop-down using SQL. However, we will need to escape the additional “backslash” (i.e. \ ) since we need MySQL to execute the query and actually generate the HTML markup that will be rendered on the browser via Javascript. Javascript does not care for arbitrary line breaks 😉 [1] and MySQL does not care about unescaped backslashes!

Now, if this sounded confusing, ponder for a moment on this Ajeet joke from yesteryears:

Raabert: Boss? Is kaa kyaa kare boss?
Ajeet: Rawbert! Is pille ko liquid oxygen me daal do. Liquid ise jeene nahi dega, aur oxygen ise marne nahi dega.

Nicole’s SQL based option list generator:
SELECT CONCAT('<option value=\"mc-itype:', itemtype, '\">',description,'</option>') FROM itemtypes

Our modification to the that make the output Javascript friendly:
SELECT CONCAT('<option value=\"mc-itype:', itemtype, '\">',description,'</option> \\') FROM itemtypes

In this case, turns out we have 5 itemtypes defined and we get this following output and we save it as CSV for introduction into our jquery.

<option value=”mc-itype:BBK”>Bengali Books</option>\
<option value=”mc-itype:BOOK”>Books</option>\
<option value=”mc-itype:BVOL”>Bound Volumes</option>\
<option value=”mc-itype:REF”>Reference Books</option>\
<option value=”mc-itype:SER”>Serials</option>\

(Hint: the exact item types is most likely to differ in your case; DO NOT copy-paste this output AS-IS.)

Let’s build the actual jQuery snippet

$( '<select name="limit" id="limitfiler" style="margin-left: 4px;"> \
<option value="">-- filter by item type --</option> \
<option value="mc-itype:BBK">Bengali Books</option> \
<option value="mc-itype:BOOK">Book</option> \
<option value="mc-itype:BVOL">Bound Volumes</option> \
<option value="mc-itype:REF">Reference Books</option> \
<option value="mc-itype:SER">Serials</option> \
</select>' ).insertAfter('#masthead_search');

And plug it into your opacuserjs system preference.

Let test our new drop-down!

add_itype_04
Fig 2: Searching for “Subject” as “English language” and no item-type based filtering
Fig 3: Additionally "BOOK" item type filter on.
Fig 3: Additionally “BOOK” item type filter on.
Fig 4: With REF itemtype filtering on.
Fig 4: With REF itemtype filtering on.

References:

[1] “Multi-Line JavaScript Strings” by David Walsh

Migration tips #1 : Separating your 100, 110 and 111s in the spreadsheet

Imagine the following scenario:  Over the last several months you have been steadily entering your library’s bibliographic data, either from the accession register or by accessing actual physical copies at your library into a spreadsheet. You have managed to create some ~23,000 records and now wish to import these into your favourite ILS i.e. Koha.

While working, in the spreadsheet you had made a single column for author information and recorded the name in the <lastname>, <firstname> style for the personal name entries (i.e. MARC21 tag 100). However you have also used the same column to capture corporate (110) or meeting (111) name entries as well.

110

So now how do you pick out these non-100 records, the 110 and 111s from among the ~23,000 records so that you do not leave any corporate body entry in the personal name field? A manual curation is possible. But it is simply too error prone and hugely time consuming.

Luckily since you have used the <lastname> comma <firstname> style consistently (or almost consistently) for all personal name entries you can use your Libreoffice Calc spreadsheet to do some magic. 😀

Step 1: A formula to identify possible corporate entries.

As we see in the example above, Calcutta University has been entered AS-IS and not as University, Calcutta so let us look for records that *do not* have any commas in them. [Sidenote: we will hit some false positives, but the magnitude of the problem will be less that searching all 23,000 records.

And for that we enter the following formula in cell B2 (where A2 is our first record and A1 being the header row):

=IF((ISBLANK(A2)),"NO",IF((ISNUMBER(FIND(",",A2))),"NO", "YES"))

The first part is just a safety check to ensure that our cell is not blank. For course, if it is blank then obviously it can not be a corporate body entry so the answer we want in B2 is “NO”. Once we know it is not indeed blank, we move in to check if we can find our “comma”  in the cell. If the comma is present then it is assumed to be a personal name entry, hence also “NO”, and if not found *and* not blank then it is safe to assume it is most likely a corporate name (110) or a meeting name (111).

Lets see what happens when we apply this formula to all our ~23,000 records.

110_1

N.B. The “false positive” happened as the person doing the data entry did not format the name of “Mahasweta Devi” as “Devi, Mahasweta”

Step 2: Always count the eggs in your basket

We had started with exactly 22,959 records, let us see how many “YES” records our formula has found: 941. We have narrowed our search down to just about 4% of the total records.

110_4

Step 3: Filtering out the “YES” records

Luckily for us, LibreOffice Calc has a nice filtering tool. It is available under Data -> Filter -> Standard filter menu option. We had earlier named our column as “boolean” in the cell B1. So we’ll now filter out all 941 records that marked as “YES” using this tool.

110_2

And immediately our spreadsheet will show us only the “YES” records like this:

110_5

Step 4: Removing the “YES” from false positives

The false positives have a story to tell. They tell us that we need to do better quality control of our data entry. We also probably need to ensure that the persons entering the data understand how to handle Muslim or foreign names.

Example: Let us take the name of my good friend and Koha expert Mr Vimal Kumar Vazhappally. Now the correct way to address him is as Mr. Vimal Kumar and *not* as Mr. Vazhappally. Vazhappally isn’t really his surname, rather it is the name of his village.

For now, the simplest way to correct the false positive is simply to visual check the A column and if it is apparent that the corresponding cell in B column wrongly has a “YES”, simply to move to the cell in B and simply delete the formula from that cell.

Instead of looking through 22,959 records, we now are going to check less than a 1000 records, but this time looking only for false positives.

110_3

Step 5: Two formula to separate the 100s and 110s

After we remove the B column cell values of the false positives – ones with “YES” but not a corporate entity, we now need two more columns to our spreadsheet. These will hold our separated 100 and 110 / 111 values.

So in order to copy over the values of corporate / meeting names (110 / 111) to the C column, we will define the following formula in the cell C2:

=IF((B2="YES"),A2,"")

And similarly in the next column to the right, in the cell D2 we will define this fomula:

=IF(B2="YES","",A2)

Next we will simply copy the range C2:D2 over to the entire range C3:D22959 and we are DONE!

110_6

Columns C and D now have our separated records for all 22959 records. And it took me less time to do the actual correction, than it took me write this blog post, take screenshots, crop, annotate, upload and proof read the final post.

Closing words

We found ~350+ corporate body / meeting name entries in the list, which could be separated out of 22,959 records using this technique.

 

MarcEdit QuickTip #2 – Unicode in your source file

Converting a batch of multi-lingual bibliographic records stored in a MS-Excel worksheet to .mrc using MarcEdit? Be sure to check your charset is set to UTF-8 while saving the spreadsheet or its CSV export.

Naveed Bhatti, a fellow Koha ILS user from neighboring Pakistan pinged last week over a problem he was facing. He had multi-lingual bibliographic data stored in an MS Excel worksheet. He wanted to use MarcEdit 6’s Delimited Text Translator tool available under the “Add-ins” menu to convert this file into an Unicode (UTF-8) encoded MARC21 (.mrc) file so that he could import the records into Koha.

However, when he attempted to generate the .mrk file (MarcEdit’s intermediate MarcBreaker mnemonic format before export to .mrc) instead of seeing the Arabic script, he saw a bunch of “?????? ????? ???” wherever there was text in the Arabic script in the spreadsheet. Naveed thought he must be missing something small but crucial, perhaps a setting.

naveed_02

I had a hunch, but I wanted to check the data before commenting. So I asked for a few sample records, which he sent over the next day. I checked and found it was a simple case of charset conversion glitch at the spreadsheet end of things. I could easily generate both the incorrect as well as the correct output (see above) with a simple change of the charset filter. I was using LibreOffice Calc on Windows 8.1 and the default export charset was *not* set to Unicode (UTF-8). The case being, with the default export charset the exported / saved file did not contain the correct Unicode codepoints in the data for the non-Latin data. As a result, at MarcEdit’s end, it became a simple case of garbage-in-garbage-out instead of receiving the correct non-Latin data.

The screenshot below shows the correct filter to use if you are using LibreOffice Calc. If you are using MS-Office, you should see something similar.

naveed_03

Downgrading your Koha ILS installation

Earlier in the day Dyuti messaged me. He was facing problems that did not exist before after he upgraded to Koha 16.05 from 3.22 on his production system. What he reported sounded like a bug, so I asked him to report it on the Koha bugzilla. Thankfully, he had remembered to take a database backup of his Koha instance before the upgrade. He wanted to know if there was any way he could go back to 3.22 using the backup where things were working fine (as far as the functions he required).

This is a fairly common scenario. People upgrading to the new stable after a major version release, often wish that they can *downgrade* to their earlier major version release, especially after realizing that something is broken in the new major version that used to work for them in the earlier one. If they could do that, then they could wait until the bug fix became available, without any interruption in the library service.

Luckily with Koha this is quite easy, especially if you were on the immediate previous version.

(HINT! one more reason why you should try to keep your Koha instance updated if you do not have a dedicated IT team to support you).

Taking Dyuti’s case as an example use-case, here are the steps to go back  to 3.22 from 16.05:

  1. Using the terminal, take a backup of koha_library database, preferably using koha-run-backups (click to reference details) and keep the backup safe.
  2. Use sudo apt-get purge koha-common command to remove Koha ILS from the system.
  3. Remove the existing koha_library database from MySQL.
  4. Re-install earlier major version series (3.22 in this case) of Koha using the oldstable repo. The change has to be made in your koha.list file (see here for explanation).
  5. Run the sudo koha-create --create-db library command to recreate your Koha instance.
  6. Import the old 3.22 backup into the newly created koha_library database.
  7. Access the Koha staff client using the new database user id and password from the newly recreated koha-conf.xml, and let the web installer run its course.
  8. Switch to a terminal and run sudo koha-rebuild-zebra -v -f library to update your zebra indexes.

You should be good to go! Enjoy!

NOTE 1: Dyuti’s Koha database is named ‘koha_library‘.  However your’s may be named differently, so remember to substitute your own db name here.

NOTE 2: These instructions are meant for a package based Koha installation either on Debian or Ubuntu as outlined in the Koha Community Wiki. If you are running a tarball or git based installation you probably already know what to do. 😀

Koha’s MARC modification templates comes to the rescue

A client had defined an item type they called “Hindi graphic books“. They used the notation “H-GR” to denote such an item e.g. an Amar Chitra Katha graphic novel. Their data was stored in a MS-Excel spreadsheet as simple tabular data. We were tasked with moving this data into Koha. So, to start off, we defined a new itemtype in Koha with the code “H-GR” with the description set to “Hindi Graphics”. The new item type correctly showed up on the OPAC advanced search under the “Item type” search filter. We then converted the data in Excel into MARC21 records using MarcEdit’s “Delimited Text Translator” tool and imported the resulting .mrc file into Koha. Import went well. However, when we searched the newly entered entries, we found that in the OPAC results page, the Item type showed up in the search facet area as “H=GR” instead of as it should – “Hindi Graphics”. Uh oh! something was certainly wrong!

itemtype_error
Figure 1: Incorrect item type displayed

Using MarcEdit we could see that this had happened due to an additional space having gotten accidentally padded to the itemtype data during data migration. To the system “H-GR” is NOT EQUAL to “H-GR “. *Every* single record in this particular batch of imported MARC records had their 942$c and 952$y fields affeccted.

itemtype_error_01
Figure 2: Extra whitespace padding highlighted.

Thankfully for us, Koha ships with the MARC modification template tool, which is basically like a “find and replace” feature on steroids! So in the end we simply defined a rule that allowed us to replace all the affected 942$c in a very simple, single batch mode operation.

itemtype_error_06A

With the template (we named it ‘correctH_GR’) in place (see above), all we now had to do was to the gather together the ids (i.e. biblionumber) of the affected records and then tell Koha to run the template against this batch of biblionumbers, using the Batch record modification tool as shown below.itemtype_error_04

After this the Batch item modification tool was used to update 952$y which is actually stored in items.itype field. And like above, with just one difference that instead of biblionumber we now need the list of barcodes for all the H-GR item type, in order to modify it.

itemtype_error_05A

And finally our original problem is solved!

itemtype_error_06

PRO-TIP: MARC modification template tool is not only useful for post-facto correction. It can be used even while we are staging MARC records in the first place, offering us to make wide-ranging changes and updates to our incoming MARC record.

Customizing Koha’s iconset to match page background

The world’s favorite open source ILS aka Koha ships with several iconsets included i.e. carredart, crystal-clear, seshat, bridge, npl, vokal etc. The primary use of these icon sets is to provide visual cues for different item types e.g. books, serials, DVDs, sound recordings, electronic materials etc. The icon set used by default is The Bridge Material Type Icon Set.
It is licensed under Creative Commons 2.5 Attribution, ShareAlike license.

This HOWTO will focus on how to customize the bridge iconset so that our icons match the OPAC page background. The iconset as shipped by Koha are in .gif format with a white background. This default works perfectly well when we stick with the default CSS of Koha, where the page background (the main div) is also white as seen below.

def_iconset

There are times when we may wish to customize the OPAC CSS using the opacusercss system preference, for instance in this case, we wanted to change the body.main div’s background-color to match our custom OPAC color theme:

/* change main div color */
div.main { background-color: #EAE9E7; }

But due to the white background of the Bridge iconset, the page now looked like this:

def_iconset_01

 

Obviously this needed to be fixed. We went to The Bridge Material Type Icon Set Project page and downloaded the zip file (downloaded from here) containing the original .psd files of the icons.

While you can use Photoshop to edit them, we used GIMP (GNU Image Manipulation Program) to edit the .psd files. We will use the book material type icon – book.psd for this example.

Instructions for editing the icons using GIMP

    1. Open book.psd in GIMP
    2. Check the layers – both visible and the ones turned off gimp_layers
    3. Set the foreground color to #EAE9E7 gimp_forecolor
    4. Add a new layer using using foreground color as set above gimp_newlayer
    5. Arrange the layers correctly with only the text layer, the graphic and the new layer using the color EAE9E7 being visible in proper order gimp_addedlayer
    6. Use GIMP’s Export As option in the File menu to create your new book.gif

Repeat the above steps until you have changed all the required icons. Save all the exported .gif files in a single directory / folder which we will call as “demolib” in this example.

Getting Koha to use the customized icons

N.B.: Koha stores the item type icons in two separate locations i.e. opac-tmpl/bootstrap/itemtypeimg/ (for the OPAC) and at intranet-tmpl/prog/img/itemtypeimg/ (for the staff client)

  1. Create an OPAC sub-folder for your customized icon set mkdir -p /usr/share/koha/opac/htdocs/opac-tmpl/bootstrap/itemtypeimg/demolib
  2. Create a staff client sub-folder for your customized icon set mkdir -p /usr/share/koha/intranet/htdocs/intranet-tmpl/prog/img/itemtypeimg/demolib
  3. Copy all the .gif files in your local demolib folder into both the newly created sub-folders above. Note: You may need to use an FTP / SCP client to copy the files, if you Koha server is located on a remote machine.
  4. Navigate to Home -> Administration -> Item types administration on your Koha system.itemtypes
  5. Click on Edit option against the itemtype Book (BK). You should now be able to see a new icon set tab named “demolib” under the “Choose an icon” section def_iconset_03
  6. Select the new icon for the Book Item type from this demolib library and save the change.
  7. Repeat for each of the item types that you wish to update with your new customized iconset.

Now visit your OPAC page again and select the “Advanced search” option. This time Koha will be using your new icons and these will merge correctly with the page background.
def_iconset_02

PRO-TIP: Instead of editing the existing Bridge icon set, you can also create your own icon set. The usual icon sizes are 32×32 pixel or 40×40 pixel (Bridge uses this dimension). Design and save them to a specific folder and then follow the instructions in the section Getting Koha to use the customized icons. Demolib is just a name we are using here, you can change it to anything else as long as they are not the ones that are already used in Koha.

Setting up a MARC21 ETD Framework in Koha

Recently during a discussion on a Whatsapp group of professional colleagues from the North East, a topic that came up – what was better suited for setting up an ETD repository at their academic institutions? As expected, most sided with DSpace, while a few suggested Eprints. I decided to introduce Koha ILS into the picture. For most, this was a rather surprising suggestion.

The ground reality

95% of Indian ETD operators do very little than scanning up a batch of printed document (bound thesis volumes) or born-digital electronic copy of the theses, make it into a PDF file, throw in some metadata together about the item and plug that into (usually) DSpace. The benefits of DSpace being statistics, organisation into collections and community (user groups), embargo capability, faceted and full text searches across the metadata. There is of course the other point of persistent URLs to the hosted resources via HANDLE system. But given that very few Indian repos actually invest in a Handle.net account, it is quite a moot point.

The argument for Koha

Utilizing MARC21, Koha already offers a highly granular and extensive metadata regime. Its capabilities include collections, search facets, full-text search on the metadata. And since Koha’s version 3.22, the capability to directly host files linked to the bibliographic records. Basically it offers everything that 95% of Indian institutions look for when they are planning to setup a repository. Compared to DSpace, LIS professionals are already better acquainted with Koha (or as many in India like to call it – “KOHA” :-P) as it is the de-facto open source LMS. Further, by using Koha for hosting the repository, the institutions and professionals who are already using Koha as their LMS, gain one key advantage; they no longer need to maintain two separate and very dissimilar systems that use completely separate software application stacks (LAMP vs Java; MySQL vs PostgreSQL etc).

Step 1 – building a MARC21 ETD Framework

I had promised my colleagues to setup a demo ETD using Koha so that they could try it out. After all, there’s no better proof of the pudding than eating it. The first challenge to that is the while Koha does ship with quite a few MARC21 frameworks, an ETD framework is not one of them.

As my starting point, I turned to ETD-MS v1.1, which is considered to be something of a gold standard for ETDs. Taking the help of this page, I worked out a single paged worksheet for cataloging ETDs  on Koha. The result looked like this:

rawmarc

Step  2- Usability and maintainability

Of course this posed a slight problem, how will a cataloger accustomed to / trained on DSpace’s metadata namespace correctly do the crosswalk? To solve this in lines with Koha’s recommended Best Practices for UI mods, jQuery statements were included into Koha’s “intranetuserjs” and “intranetusercss” system preferences. The final outcome was this:

converted_marc

This approach has three (03) key benefits  – (a) that you DO NOT need to touch Koha’s Template::Toolkit based templates (i.e. the .tt files), the changes made are stored in your database and applied during runtime; (b) these changes remain persistent and works across Koha’s monthly version upgrade cycle (since we didn’t change the .tt files) and (c) our other MARC21 templates are left alone, only the ETD framework is thus modified.

The jquery snippet is available on Github as a gist, as is the CSS includes into intranetusercss system preference. There is also a third gist which lists the 04 authorised value categories and their constituent sample options that help us introduce a consistent controlled vocabulary in our metadata.

The demo ETD is available at http://etddemo-opac.l2c2.co.in/. If you wish to access the staff client back-end, contact me via the comments section of this post.

Best wishes!

Resources:

  1. http://blog.l2c2.co.in/wp-content/uploads/2016/06/export_ETD.ods
  2. https://gist.github.com/l2c2technologies/09e7e06f695304f33aada9b529167de6
  3. https://gist.github.com/l2c2technologies/9bc1f9b812e37850959c655fbc0f8802
  4. https://gist.github.com/l2c2technologies/6b735a5e1f4c4f9cc3042af2b8fa5b32
Licensing- CC 4.0 – BY-SA-NC – (c) L2C2 Technologies 2016

The Koha manual is your friend :-)

The Koha User Manual is an amazingly in-depth document that keeps getting better with each iteration, thanks to Koha Library System Project’s indefatigable Documentation Manager Nicole C. Engard. This post exemplifies a situation where it can come in handy.

Yesterday a librarian colleague reached out with a query. He is on Koha 3.20.x series and since the last 3 – 4 days, he had been noticing that the “due date” for items his staff was issuing out was coming out rather odd! For instance all the due dates were set to “18th June 2016“. He clarified that he did not have a hard due date set, so what is going on???

This morning he sent over a screen shot of the checkout operation. And boy! the date now read 25th-June-2016. This was an immediate clue! On checking out his calendar – 18th and 25th turned out to be the only two days that are marked as “library is open” in June 2016 (for issues made in these last two weeks i.e.  before the holidays begin) during their summer recess.

That holiday setting working together with the system preference “useDaysMode” set to “the calendar to push the due date to the next open day” for circulations was the reason why Koha was pushing the due date beyond the usual designated due dates for each patron category.  Basically, it was just Koha doing its job (as it was asked to do) and doing it well.

If you are on 3.20, remember to read up http://manual.koha-community.org/3.20/en/calholidays.html, the relevant system preference are all explained very nicely there by Nicole C. Engard, Koha’s indefatigable Documentation Manager.