Remote mirroring between two secured MapR cluster

AIM

To setup remote mirroring between two secured MapR cluster. (In the example given below both source and destination cluster has single node.)

In source Cluster:

Cluster name: ajames

Securing the source cluster

/opt/mapr/server/configure.sh -C <source_cluster> -Z <source_cluster> -N ajames -secure -genkeys

Check the cluster conf to verify cluster is secured

[root@c51-4 ~]# cat /opt/mapr/conf/mapr-clusters.conf

ajames secure=true <source_cluster>:7222

Change the permission of ‘ssl_keystore’ to 600

chmod 600 ssl_keystore

Start the zookeeper and warden services

service mapr-zookeeper start

service mapr-warden start

Login as mapr user

su mapr

Generate the maprlogin password

maprlogin password

maprcli node list -columns svc

[mapr@c51-4 conf]$ maprcli node list -columns svc

service                              hostname  ip

webserver,cldb,fileserver,hoststats  c51-4     <source_cluster> 

In destination Cluster:

Cluster name: ajames2

Securing the destination cluster

/opt/mapr/server/configure.sh -C <destination_cluster> -Z <destination_cluster> -N ajames2 -secure -genkeys

Check the cluster conf to verify cluster is secured

[root@c51-4 ~]# cat /opt/mapr/conf/mapr-clusters.conf

ajames2 secure=true <destination_cluster>:7222

Change the permission of ‘ssl_keystore’ to 600

chmod 600 ssl_keystore

Start the zookeeper and warden services

service mapr-zookeeper start

service mapr-warden start

Login as mapr user

su mapr

Generate the maprlogin password

maprlogin password

[mapr@c51-5 root]$  maprcli node list -columns svc

service                              hostname  ip

webserver,cldb,fileserver,hoststats  c51-5     <destination_cluster>

In source cluster:

Merging ssl_truststore

Copy the ssl_truststore from source cluster to ‘/tmp’ of destination cluster.

scp /opt/mapr/conf/ssl_truststore root@<destination_cluster>:/tmp/

In destination cluster

As ‘root’ user

cd /tmp

chown mapr:mapr ssl_truststore

As ‘mapr’ user

/opt/mapr/server/manageSSLKeys.sh merge /tmp/ssl_truststore /opt/mapr/conf/ssl_truststore

cd /opt/mapr/conf

Add ‘ajames secure=true <source_cluster>:7222’ (details of source cluster) in

vim mapr-clusters.conf

cat mapr-clusters.conf

ajames2 secure=true <destination_cluster>:7222

ajames secure=true <source_cluster>:7222

Restart the ‘webserver’

maprcli node services -action restart -name webserver -nodes <destination_cluster>

In the source cluster

cd /opt/mapr/conf

Add ‘ajames2 secure=true <destination_cluster>:7222’ (details of source cluster) in

vim mapr-clusters.conf

cat mapr-clusters.conf

ajames secure=true <source_cluster>:7222

ajames2 secure=true <destination_cluster>:7222

Restart the ‘webserver’

maprcli node services -action restart -name webserver -nodes <source_cluster>

In source cluster:

Create the ‘crosscluster’ ticket

maprcli security getmaprclusterticket -clusterusername mapr -inmaprserverticketfile /opt/mapr/conf/maprserverticket -ticketfile /opt/mapr/conf/maprclusterticket

In destination cluster:

Copy and append the ticket to the destination cluster in maprserverticket

maprlogin password -cluster ajames

Check whether both clusters are accessible from the destination cluster

hadoop fs -ls /mapr/ajames

[mapr@c51-5 root]$ hadoop fs -ls /mapr/ajames

Found 8 items

drwxr-xr-x   - mapr mapr          0 2016-07-25 19:28 /mapr/ajames/apps

drwxr-xr-x   - mapr mapr          0 2016-07-25 19:27 /mapr/ajames/hbase

drwxr-xr-x   - mapr mapr          0 2016-07-25 19:29 /mapr/ajames/opt

drwxrwxrwx   - mapr mapr          0 2016-07-25 19:27 /mapr/ajames/tmp

drwxr-xr-x   - mapr mapr          0 2016-07-25 19:29 /mapr/ajames/user

drwxr-xr-x   - mapr mapr          1 2016-07-25 19:28 /mapr/ajames/var

hadoop fs -ls /mapr/ajames2

[mapr@c51-5 root]$ hadoop fs -ls /mapr/ajames2

Found 9 items

drwxr-xr-x   - mapr mapr          0 2016-07-25 21:09 /mapr/ajames2/apps

drwxr-xr-x   - mapr mapr          0 2016-07-25 21:09 /mapr/ajames2/hbase

drwxr-xr-x   - mapr mapr          0 2016-07-25 21:11 /mapr/ajames2/opt

drwxrwxrwx   - mapr mapr          0 2016-07-25 21:09 /mapr/ajames2/tmp

drwxr-xr-x   - mapr mapr          0 2016-07-25 21:11 /mapr/ajames2/user

drwxr-xr-x   - mapr mapr          1 2016-07-25 21:10 /mapr/ajames2/var

In source cluster:

Create a volume at source cluster ‘test1’

Create a file in the volume

hadoop fs -touchz /test/abc.txt

 

In destination cluster:

From destination cluster create the remote mirror from CLI

Example: maprcli volume create -name testmirror -path /testmirror -type mirror -source test@ajames -quota 50G -advisoryquota 40G -schedule 3 -mirrorschedule 3 -topology /data

Login to MCS and start mirroring to test the same.

Sqoop – Tibero Integration

Version:

Sqoop  - 1.4.1

Tibero - 6

Tibero installation in UNIX system:

Prepare the following two files. (You can get trial edition in http://www.tmaxsoft.com/)

·        Binary executable file (tar.gz)

·        License file (license.xml)

Extract the tar:

tar -xvf tibero5-bin-5_rel_FS05-linux64-127068-opt-tested.tar.gz

Create a folder ‘license’ in the tibero home and copy the license.xml

Execute the following:

./Tibero_6_FS04_linux6420160418.bin

Check that the environment variables are properly set in the environment configuration files (.bashrc, .bash_profile, .profile, etc.) for each user account for each OS. Sample given below:

#Created by Tibero Installer. You should manually delete below variables for uninstall.

export TB_HOME=/root/TmaxData/tibero6

export TB_SID=tibero

export PATH=/usr/xpg4/bin:/usr/xpg4/bin:/sbin:/bin:/usr/sbin:/usr/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/root/TmaxData/tibero6/bin:/root/TmaxData/tibero6/client/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/root/TmaxData/tibero6/bin:/root/TmaxData/tibero6/client/bin

export LD_LIBRARY_PATH=/root/TmaxData/tibero6/lib:/root/TmaxData/tibero6/client/lib

###############################################################################

 

Execute the following command in $TB_HOME/config.

./gen_tip.sh

 

Start the database in ‘NOMOUNT’ mode from $TB_HOME:

bin/tbboot nomount

Create a database using the CREATE DATABASE statement. (Very basic)

create database "tibero"

  user sys identified by tibero;

Restart Tibero using the tbboot command.

Execute system.sh command in $TB_HOME/scripts to execute the SQL file in the current directory. (The default passwords for the sys and syscat account are tibero and syscat, respectively.)

Start the tibero database client:

client/bin/tbsql sys/tibero

 

Checking the tibero services

[root@c51-3 ~]# ps -ef | grep tbsvr

root     28167     1  0 Jul18 ?        00:04:10 tbsvr          -t NORMAL -SVR_SID tibero

root     28169 28167  0 Jul18 ?        00:00:00 tbsvr_TBMP     -t NORMAL -SVR_SID tibero

root     28170 28167  0 Jul18 ?        00:00:00 tbsvr_WP000    -t NORMAL -SVR_SID tibero

root     28171 28167  0 Jul18 ?        00:00:00 tbsvr_WP001    -t NORMAL -SVR_SID tibero

root     28172 28167  0 Jul18 ?        00:39:37 tbsvr_AGNT     -t NORMAL -SVR_SID tibero

root     28173 28167  0 Jul18 ?        00:10:02 tbsvr_DBWR     -t NORMAL -SVR_SID tibero

root     28174 28167  0 Jul18 ?        00:00:00 tbsvr_RECO     -t NORMAL -SVR_SID tibero

root     31398 17165  0 22:39 pts/1    00:00:00 grep tbsvr

Integrating Sqoop

Copy tibero6-jdbc-14.jar to /opt/mapr/sqoop/sqoop-1.4.6/lib/

Importing data from Tibero:

sqoop import --driver com.tmax.tibero.jdbc.TbDriver --connect jdbc:tibero:thin:@localhost:8629:tibero --username sys --password tibero --table emp -m 1;