When a 12c upgrade unexpectedly breaks something - beware 9i!



The 12c upgrades continue to rumble on - this week we upgraded our cloud control repository (Oracle finally added support to be able to do this) - this all went fine however a couple of days later we had reports that the 'portal' isn't working.

Now the 'portal' is an ancient utility system developed in house that was considered to be unused (at least by us) - however it appeared that some user groups were still using it. It's nothing to do with the cloud control setup it just shares the database.

This was erroring with

ORA-28040: No matching authentication protocol

A quick look at the error revealed that this (which i'm sure will become quite a familiar error to everyone) is due to us trying to use a 9i client to connect to 12c - this is not supported at all - the earlier version you can use is 10g.

Right - let's have a look at the setup of the portal to see how we can fix this.

A quick look reveals that ORACLE_HOME etc is explicitly set in the environment before the application is started (the application being apache/perl) - so we just change that to the 10g home that is installed on the box and restart and it's fixed right?

Well sort of - some of the screens seem to be using the generic db connectivity from perl and do start to work with this change - however a couple of screens use the perl DBD module for oracle - these are not working....

After a bit more digging we discovered this

dump -H ./Oracle.so

./Oracle.so:

                        ***Loader Section***
                      Loader Header Information
VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x000000f2 0x000001d9 0x00000092

#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000004 0x00002cfc 0x0000103b 0x00002d8e


                        ***Import File Strings***
INDEX PATH BASE MEMBER
0 /usr/local/lib:/app/oracle/product/9.2.0.5/lib32/:/app/oracle/product/9.2.0.5/rdbms//lib32/:/usr/lib:/lib 
1 libc.a shr.o
2 libclntsh.a shr.o


So the module looks for libclntsh.a  only in the paths above - it's not considering any environment variables.

So how do we fix that?

Luckily the first path listed is /usr/local/lib so we can just create a sym link from this directory to the 10g home and it will find that first and fix the problem.

We try that and refresh the screen and get this

install_driver(Oracle) failed: Can't load '/portal/perl/lib/site_perl/5.8.6/aix/auto/DBD/Oracle/Oracle.so' for module DBD::Oracle: Could not load module /portal/perl/lib/site_perl/5.8.6/aix/auto/DBD/Oracle/Oracle.so.
  Dependent module /usr/local/lib/libclntsh.a(shr.o) could not be loaded.
  The module has an invalid magic number.
System error: Exec format error
Could not load module /usr/local/lib/libclntsh.a.

So progress - however i've sym linked to the wrong files - i pointed at the 64 bit ones and perl is only 32bit on this server.

we finally get the command right and run this

ln -s /app/oracle/product/10.2.0.2.DB/lib32/libclntsh.a /usr/local/lib/libclntsh.a 

And the problem is fixed!

So a relatively easy fix in this case - but a mjor gotcha with 12c upgrades is that 9i cannot connect at all - bear this in mind when doing any upgrades - i'm sure there is still a lot of 9i out there....



Comments