Tuesday, March 23, 2010

Datastage Repository - XMETA

Prior to Datastage version 8, the datastage repository information was stored in files(uv). From version 8, the repository information is also stored in database. Since the information is available in database, it is easier to write queries and find the details. During the installation you will have the option to install the repository either in DB2 or in Oracle. Currently there is not much docuementation about the XMETA repository from IBM.


In my project, the datastage repository was built on DB2 database. In fact, datastage installation comes with DB2 database software.
I found that the table names and column names are bit weird with some funky characters & numerals at the end may be that's IBM strategy to keep this information away from the developers..:). Please note that table names listed below may not be exactly same for you as the characters at the end of the table name may be different for each installation.


A word of caution: Do not try to alter the XMETA repository contents as it may have adverse effects.


Below are some of the queries that I have developed and I found that useful
1. Query to list Projects
SELECT *
FROM
         XMETA.DATASTAGEX_XMETAGEN_DSPROJECTC2E76D84;


2. List the folders within a project
SELECT *
FROM
         XMETA.DATASTAGEX_XMETAGEN_DSFOLDERC2E76D84
WHERE
         XMETA_CREATED_BY_USER_XMETA != 'DataStageSystemUser'
AND NAMESPACE_XMETA = 'myproject';


3. Query to retrieve the DS JOB information
SELECT
(TIMESTAMP('01/01/1970', '00:00:00') + (XMETA_CREATION_TIMESTAMP_XMETA / 1000) SECONDS) AS CREATION_TIME,
(TIMESTAMP('01/01/1970', '00:00:00') + (XMETAMODIFICATIONTIMESTAMPXMET / 1000) SECONDS) AS MODIFIED_TIME,
A.*
FROM
         XMETA.DATASTAGEX_XMETAGEN_DSJOBDEFC2E76D84 AS A
WHERE
         NAME_XMETA = 'mydsjob';


4. Query to list the jobs that are locked
SELECT
A.NAME_XMETA, B.LOCKED_BY_USER,
(TIMESTAMP('01/01/1970', '00:00:00') + (B.LOCKED_AT / 1000) SECONDS) AS LOCKED_AT
FROM
        XMETA.XMETALOCKINFO B,
        XMETA.DATASTAGEX_XMETAGEN_DSJOBDEFC2E76D84 A
WHERE
        A.XMETA_REPOS_OBJECT_ID_XMETA = B.REPOSITORY_ID;


5. Query to list the stages within the job
SELECT A.NAME_XMETA JOB_NAME, S.NAME_XMETA STAGE_NAME,
S.STAGETYPE_XMETA STTAGE_TYPE,S.*
FROM
        XMETA.DATASTAGEX_XMETAGEN_DSSTAGEC2E76D84 S,
        XMETA.DATASTAGEX_XMETAGEN_DSJOBDEFC2E76D84 A
WHERE
        S.CONTAINER_RID = A.XMETA_REPOS_OBJECT_ID_XMETA
AND A.NAME_XMETA = 'mydsjob';


6. Query to show the job details by folder
SELECT CONTAINER_RID, XMETA_REPOS_OBJECT_ID_XMETA, XMETA_CREATED_BY_USER_XMETA,
(TIMESTAMP('01/01/1970', '00:00:00') + (XMETA_CREATION_TIMESTAMP_XMETA / 1000) SECONDS) AS XMETA_CREATION_TIMESTAMP_XMETA , XMETA_MODIFIED_BY_USER_XMETA, (TIMESTAMP('01/01/1970', '00:00:00') + (XMETAMODIFICATIONTIMESTAMPXMET / 1000) SECONDS) AS XMETAMODIFICATIONTIMESTAMPXMET,NAME_XMETA,SHORTDESCRIPTION_XMETA,
JOBTYPE_XMETA,CATEGORY_XMETA,DSNAMESPACE_XMETA FROM
        DATASTAGEX_XMETAGEN_DSJOBDEFC2E76D84
WHERE 
        upper(DSNAMESPACE_XMETA) = upper('myproject')
and upper(category_xmeta) like upper('\\Jobs\\');

7. List all the table definitions
SELECT
(TIMESTAMP('01/01/1970', '00:00:00') + (XMETA_CREATION_TIMESTAMP_XMETA / 1000) SECONDS) - 7 HOURS AS CREATION_TIME,
(TIMESTAMP('01/01/1970', '00:00:00') + (XMETAMODIFICATIONTIMESTAMPXMET / 1000) SECONDS) - 7 HOURS AS MODIFIED_TIME,
T.*
FROM
           XMETA.DATASTAGEX_XMETAGEN_DSTABLEDEFINITIONC2E76D84 T
WHERE
           XMETA_CREATED_BY_USER_XMETA != 'DataStageSystemUser'
AND DSNAMESPACE_XMETA = 'myproject'
AND NAME_XMETA = 'mydsjob';

Note:
1. Timestamp is stored in UNIX Time or POSIX time format.(http://en.wikipedia.org/wiki/Unix_time).
2. In the queries above, timestamp is output in GMT.


I will update the blog when i find time to do more research on this.

Tuesday, March 16, 2010

Datastage Routines for Packed Decimal & Signed Integer


In our project, we have a requirement to send data to Mainframe System from datastage. We were having issues creating packed decimal(comp-3) and signed integer.

To resolve this, we have written 2 routines to perform the following
  • Convert integer to Packed Decimal(Comp-3)
  • Convert integer to signed integer
We realized that we cannot directly use the ICONV function with MP tag since it only works for the positive integer.So we had to manipulate the last nibble to get the correct value.

If you are interested here's the complete code of the routines:

Routine 1: Integer to Packed Decimal
This Routine converts Integer to packed-decimal value
It takes 2 Arguments
  1. inIntValue -> Input integer value
  2. inLength -> Input Length of the unpacked decimal field
Routine Code:
$INCLUDE DSINCLUDE JOBCONTROL.H
*** Declaration ***
vIntValue=inIntValue
vUnpackLength=inLength
vPackLength=int(((vUnpackLength +1)/2)+.5)
vRevSign=0
vZero=Iconv(00,"MP")

If vIntValue >= 0 Then vRevSign=1; vIntValue=Neg(vIntValue)
Else vRevSign=0

vPackDeciValue=iconv(vIntValue,"MP")

*** Pad leading zeroes
A=0
Loop
vZero=vZero:Iconv(00,"MP")
A=A+1
until A=vPackLength
Repeat

vPackDeciValue=right(vZero:vPackDeciValue,vPackLength)
*vPackDeciValue=vZero:vPackDeciValue

*** Manipulating the last nibble based on the sign
vLastByte=right(vPackDeciValue,1)

Begin case
Case vRevSign=1
vLastByte=char(vLastByte - 1)
End Case

*** To output zero as positive **
If inIntValue=0 Then vLastByte=char(12)
outPackDeciValue=left(vPackDeciValue,vPackLength-1):vLastByte
Ans=outPackDeciValue


Routine 2: Integer to Signed Integer
This Routine converts Integer to Signed Integer value
It takes 2 Arguments
  1. inIntValue -> Input Integer value
  2. inLength -> Input Length of the field without the sign
Routine Code:
$INCLUDE DSINCLUDE JOBCONTROL.H
*** Declaration
vZero=0

*** Generate zeroes
A=0
Loop
vZero=vZero:0
A=A+1
until A=inLength
Repeat

If inIntValue>=0 Then vRevSign=1
Else vRevSign=0

vIntPositiveData = ABS(inIntValue)

*** Pad Zeroes
vIntPositiveData = Right(vZero:vIntPositiveData, inLength)

vIntData = LEFT(vIntPositiveData, inLength -1)
vLastByte = RIGHT(vIntPositiveData , 1)

*** Pack the sign into the last byte
Begin case
Case vRevSign=1
vLastByteHex = Char(192 + vLastByte)
Case vRevSign=0
vLastByteHex = Char(208 + vLastByte)
End Case

outSignedIntValue = EBCDIC(vIntData ):vLastByteHex
Ans = outSignedIntValue