Salem's Euphoria

Sharing Experience


Leave a comment

Ambari – Remove a Host

I hope this will get easier with Ambari next time. The easiest way to remove a host server1.domain.net from an Ambari Cluster (ambari.domain.net) right now is:

1 – Run this curl command to get installed services on this host (admin:admin is the username:password you were using to access your Ambari GUI, cluster_name is the name of your cluster).

curl -u admin:admin -H "X-Requested-By: ambari" -X GET http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.fr/host_components | grep host_components

The result will look like :

"href" : "http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/HBASE_CLIENT",
"href" : "http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/HCAT",
"href" : "http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/HDFS_CLIENT",
"href" : "http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/HIVE_CLIENT",
"href" : "http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/HIVE_METASTORE",
"href" : "http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/HIVE_SERVER",
"href" : "http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/HST_AGENT",
"href" : "http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/INFRA_SOLR_CLIENT",
"href" : "http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/KNOX_GATEWAY",
"href" : "http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/MAPREDUCE2_CLIENT",
...
"href" : "http://ambari.domain.net:8080/api/v1/clusters/bopam_clu/hosts/server1.domain.net/host_components/METRICS_MONITOR",
"href" : "http://ambari.domain.net:8080/api/v1/clusters/bopam_clu/hosts/server1.domain.net/host_components/MYSQL_SERVER",

2 – Delete the listed services by issuing the following curl command for each SERVICE NAME :

curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/SERVICE_NAME

Ex :

curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net/host_components/HBASE_CLIENT

3 – You can now delete the host:

curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://ambari.domain.net:8080/api/v1/clusters/cluster_name/hosts/server1.domain.net

 

4 – Check the rest of the services depending on this host.

We are always still waiting for the Ambari multi-cluster support and the famous multi-everything architecture from HortonWorks team.

Advertisements


Leave a comment

Implicit updateRequestProcessorChain call – Solr

How to concat two fields in schemaless mode with Solr in Cloud mode?

1 – Create a js script file (concat_fields.js) and edit the following code:


function processAdd(cmd) {
doc = cmd.solrDoc;
id = doc.getFieldValue("id");
val1= doc.getField('field1').getValue();
val2 = doc.getField('field2').getValue();
separator = params.get('separator');
doc.setField("field3", val1+separator+val2);
}

Continue reading


Leave a comment

Ambari uninstall scripts

 

If you want to remove an Ambari-through install of HDP components, you will have to it manually.

Actually, Ambari gives you a way to uninstall services and remove hosts. But, this feature assumes that you have completed the components install process successfully. What if the process fails somewhere when installing?

In my case, I collected all the commands from many posts and grouped them into one single script. You can find the script in my github.

By the way, this is a very well written manual on how to setup HDP 2.5.


Leave a comment

Setup HBase Indexer (Part 2)

1 – Why would someone use Solr to search on a wide-column database (HBase)?

The power of HBase search (scans) is not filters. All is about the rowkey design. If you want to take full advantage of HBase, you must know all your search queries at the moment of deigning your database. This way, you will put all the “search” intelligence in your rowkeys. But what if you don’t know all your search criteria at the beginning? What if you need to add extra search criterias? Would you create a new “view” of data with another rowkey strategy? What would you do if your client needs to search by “proximity” or a did you mean style?

There is no answer for this question than “it depends”.

 

2 – Why we did not use Ambari for Solr deployment?

It is not integrated offcially, it does not bring any added-value, it adds some more complexity in ambari-agents scripts (must be altered manually for this use case).

Continue reading


Leave a comment

Setup HBase Indexer (Part 1)

Pre-requisites:

The scope of this post does not cover Hadoop/Hbase setup. I asume that you have a running Hbase environment with a Master (HMaster) and two region servers (rs1 and rs2).

I’ll be using the HDP2.5 release from HortonWorks setup on CentOS 7.2.

1 – Setup Solr

Actually, I don’t want Ambari to manage my Solr instance because, we have some specific configurations to add and we won’t alter default ambari-agent’s behaviour.

sudo rpm --import http://public-repo-1.hortonworks.com/HDP-SOLR-2.5-100/repos/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
sudo cd /etc/yum.repos.d/
sudo wget http://public-repo-1.hortonworks.com/HDP-SOLR-2.5-100/repos/centos7/hdp-solr.repo
sudo yum install lucidworks-hdpsearch

Continue reading


Leave a comment

HBase, Zookeeper and Solr

 

If Solr and HBase are not on the same machine (distributed architecture), you will probably face this ZK problem:

[WARN ][15:14:42,409][host:2181)] org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
 at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

java.io.IOException: Failed to connect with Zookeeper within timeout 30000, 
connection string: localhost:2181
 at com.ngdata.hbaseindexer.util.zookeeper.StateWatchingZooKeeper.<init>(StateWatchingZooKeeper.java:109)
 at com.ngdata.hbaseindexer.util.zookeeper.StateWatchingZooKeeper.<init>(StateWatchingZooKeeper.java:73)
 at com.ngdata.hbaseindexer.cli.BaseIndexCli.connectWithZooKeeper(BaseIndexCli.java:92)
 at com.ngdata.hbaseindexer.cli.BaseIndexCli.run(BaseIndexCli.java:79)
 at com.ngdata.hbaseindexer.cli.AddIndexerCli.run(AddIndexerCli.java:50)
 at com.ngdata.hbaseindexer.cli.BaseCli.run(BaseCli.java:69)
 at com.ngdata.hbaseindexer.cli.AddIndexerCli.main(AddIndexerCli.java:30)

Notice here that HBase-indexer is trying to reach the localhost server of Zookeeper.
Do not to forget –zookeeper param in distributed HBase-Indexer setup:
/opt/lucidworks-hdpsearch/hbase-indexer/bin/hbase-indexer add-indexer -n hbaseindexer -c /opt/lucidworks-hdpsearch/hbase-indexer/indexdemo-indexer.xml -cp solr.zk=hmaster:2181 -cp solr.collection=hbaseCollection –zookeeper hmaster:2181

Continue reading


Leave a comment

CentOS 7.2 VM hangs on startup

Host :  Windows 10
Guest : CentOS 7.2 64x
Tool : Oracle VirtualBox 5.2

Problem:

For an unknown reason, your VM won’t startup and hangs on boot. You have tried like me many VBoxManage commands, transform raw to vdi, file access rights, … but you’re still on this screen:

CentOS hangs here

 

Solution :

When your machine gives you the boot choice, press “e” to enter the grub configuration. Look for “rhgb” parameter in the kernel boot commands and delete it.

2017-06-26_1049

This command should be in the line starting with “linux16”.

Add the command “systemd.unit=multi-user.target” at the end of this line.

Press Ctrl+x to start the VM and maybe everything will work fine.