CentOS hangs on startup after yum update

CentOS

Usually it will hang after Starting certmonger: [OK]

Startup the hanging machine (H1)
From another machine do:

> ssh H1
> su
> mv /etc/X11/xorg.conf /etc/X11/xorg.conf.bak

Now restart H1.

You can have more explanations by doing a “tail -n 200 /var/logXorg.0.log” on the hanging machine.

How to remove duplicate lines in MySql in one command!

This simple mysql command in InnonDB engine may save you 30 minutes if you want to remove the duplicate copies of a row. Consider a table called “your_table_name” having many columns : column1, column2, column3, …. You define a duplicate row as a row having the same values as in column1 and column2. You have just to create a unique index based on those columns.

ALTER IGNORE TABLE your_table_name ADD UNIQUE INDEX give_ur_index_a_name (column1, column2 );

Commit, and check your table again… duplicate rows went away!

Update: In some versions of MySQL, “ALTER IGNORE” won’t ignore the duplicate key problem. So you may have to run the following command :

set session old_alter_table=1;

After adding the index, do not forget to set old_alter_table to “0” again.

2014 in review

The WordPress.com stats helper monkeys prepared a 2014 annual report for this blog.

Here’s an excerpt:

A New York City subway train holds 1,200 people. This blog was viewed about 3,800 times in 2014. If it were a NYC subway train, it would take about 3 trips to carry that many people.

Click here to see the complete report.

Just another mirror for HBase

While testing the Hadoop database, HBase, I noticed that dedicated mirrors were very slow. Here’s another link using 4shared.

hbase-0.99.2-bin.tar.gz

 

By the way, if you see this error while trying to start HBase (./start-hbase.sh):

$ ./start-hbase.sh
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/hadoop/hbase/util/HBaseConfTool : Unsupported major.minor version 51.0

Be sure that you’re running the minimum required JDK (1.7) and that the HBASE_HOME is set. I found this line in hbase-env.sh:

# The java implementation to use.  Java 1.7+ required.
# export JAVA_HOME=/usr/java/jdk1.6.0/

Simple JTAPI Applet – CISCO Implementation

JTAPI is a Java library intended to ensure communication between a Java client and a CTI server. There are many implementations of JTAPI, proprietary(AVAYA, CISCO,…) and open source(GJTAPI, XTAPI). I’ve been working on an Applet to replace the CISCO IP Communicator. This applet uses the CISCO JTAPI implementation to listen to incoming calls, emit consultation calls and execute a call transfer. A consultation call is a call to a target number (B) started by an agent while receiving another call from source (A). The active call from A is switched to “waiting mode” (listening to some music on the wait tone). The agent has the possibility to join source A to target B without listening to their conversation (not a conference).

To achieve this task, I’ve been reading the JTAPI developer guide. The test platform preparation is not discussed in this post. Two common problems to solve to make this applet work correctly: How to make the applet access to local resources(mainly client logging)? How to implement these functions with such a restricted community?

To enable the applet interaction with client local resources, I implemented a test one. the applet looks like :

<div>
<applet id="sph" code="com.interop.cisco.MiniSoftphone.class"
archive="miniphone.jar,jtapi.jar,log4j.jar" />

</div>

The miniphone.jar holds my implementation of the applet (MiniSoftphone.java), “jtapi.jar” is the cisco JTAPI implementation and “log4j.jar” is the default logging library for CISCO JTAPI. Note that this applet also interacts with an SWF (the main application) via some JS code. To avoid paths problems, I moved those libraries to the root context of my application under the server.

To execute the applet in IE for example, you have first of all to sign you jar with a valid certificate (using jarsigner tool from the default JDK installation bin). You will have to update your browser plugin  to the latest java release. Next, you have to trust your host in the java configuration panel.

Trust Server

To manage logging params in JTAPI, use the given CISCO tool(Cisco Unified Communications Manager JTAPI Preferences). You need also to place the jtapi.ini side by side with jtapi.jar. If this file is not generated during installation process, use the following command:

java CiscoJtapiVersion -parms

Then you can manage logging properties. It is useful to enable all traces for tests for easier troubleshooting.

2014-12-04_1514Logging management

 

 

 

 

 

 

 

 

 

 


 

In the Log Destination tab, I used “c:\TEMP\” as logging path. This will create a rolling file where every action of the JTAPI is logged. You can also use the client java console as an output. An interesting tip here, is how to print an exception stack trace into a given text box for example (inConsole is an AWT TextField in this snippet).

  StringWriter sw = new StringWriter();
  PrintWriter ps = new PrintWriter(sw);
  e.printStackTrace(ps);
  inConsole.append(sw.toString()+"\r\n");

One more thing to add in the user home to enable the applet write in a local directory: Create a file called “.java.policy” in the user home dir, and assign wanted rights to your source server:

grant codeBase "http://localhost:8080/ctiapp/*" {
  permission java.io.FilePermission "C:\\TEMP\\*", "read";
};
grant codeBase "http://localhost:8080/ctiapp/*" {
  permission java.io.FilePermission "C:\\TEMP\\*", "write";
};

Now, as soon as I went to my applet web page,  IE asked me to accept this “hazardous” applet:

Applet warning signed jar

So, I’m done with the first problem. Let’s now focus on the JTAPI implementation.

First of all, we need a valid CUCM server with valid credentials. We have to implement the ProviderObserver interface with its providerChangedEvent (aProviderObserverImpl).


public void providerChangedEvent ( ProvEv [] eventList ) {
if ( eventList != null ) {
for ( int i = 0; i < eventList.length; i++ )
{
if ( eventList[i] instanceof ProvInServiceEv )

{
conditionInService.set ();
}
}
}
}

Then, we try to connect the applet to the CUCM (providerName=CUCM IP; Login=cucem username; passwd=cucm password):

 Provider provider;
 JtapiPeer peer = JtapiPeerFactory.getJtapiPeer ( null );
 String providerString = providerName + ";login=" + login + ";passwd=" + passwd;
 logger.info( "INFCTI101 Trying to reach provider[Connection String :" + providerString + "]" );
 provider = peer.getProvider ( providerString );
 provider.addObserver ( aProviderObserverImpl);
 conditionInService.waitTrue ();

Next, we need to implement the CallObserver interface to listen to call events.

protected final void metaEvent ( CallEv [] eventList ) {
 //logger.info("INFCTI100 Received a CallEv array event - Length = "+eventList.length);
 for ( int i = 0; i < eventList.length; i++ ) {
 TerminalConnection tc = null;
 try {
 CallEv curEv = eventList[i];
 // logger.info("INFCTI101 Processing event item : "+i);
 if ( curEv instanceof CallCtlTermConnRingingEv ) {
 // logger.info("INFCTI102 Event type is : CallCtlTermConnRingingEv.");
 tc = ((CallCtlTermConnRingingEv)curEv).getTerminalConnection ();
 //logger.info("INFCTI103 CTI connection is : "+(tc!=null?tc.getState():"null"));
 tc.answer ();
 //logger.info("INFCTI104 Agent answered the call....");
 }else if (curEv instanceof CallControlCall) {
 CallControlCall ccc = (CallControlCall)curEv.getCall();
 String callerName = ccc.getCallingAddress().getName();
 System.out.println("Received a call from "+callerName);
 //logger.info("INFCTI105 Event type is : CallControlCall. Caller =["+callerName+"]");
 }
 }
 catch ( Exception e ) {
//redirect the e.printstacktrace to a textField for example
 }
 }
 }

//still a draft…

A very short introduction to MongoDB

Mong

 

Originally called humongous (big, enormous), the MongoDB project is carried by Eliot & Dwight since 2007. This is one of the best NoSql concepts implementation. For SQL fanatics, there is a small comparison between the SQL and the MongoDB NoSQL implementation.

NoSQL does not stand for “NO TO SQL” or “NON SQL”. It stands for “Not Only SQL”.

Recently I was testing it and I decided to pin this post for me and my team as a reminder.

MongoDB does not offer any official GUI. They delegated the mission to the community, and I think it was the right choice as there are now many GUI tools developed by the growing MongoDB community.

You can download the latest version from the mongodb.org. Install it as an administrator if you are on Windows.

The default datastore for MongoDB is c:\data\db. Alternatively, you can specify the datastore when starting the DB engine (–dpath=”yourpath”).

To start the MongoDB server, run “mongod” on CMD. By default, 27017. You can check if the server was started by taping this address in the browser http://localhost:27017/. You will get this message on the page (It looks like you are trying to access MongoDB over HTTP on the native driver port.)
Open another console, and run “mongo.exe”.  First thing you must be aware of, there’s no “CREATE DATABASE” command on MongoDB. Let’s now create a new database called “knw”.  We’ll create a “table” called “projects” inside this database. A table in the MongoDB terminology is called a collection. A collection is a set of “rows” (called documents in this NoSQL variant). We’ll back to collections later in this post.

 >mongo.exe
 MongoDB shell version: 2.6.3
 connecting to: test
//something like mysql use to specify current database
 > use knw 
 switched to db knw
 > show dbs;
 admin (empty)
 local 0.078GB
//there are no custom databases yet
//now, we'll insert a document in the table projects (not yet created)
 > db.projects.insert({name:"DALEEL", 
                  description:"Directories and VAS plateforms for call centers", 
                  start_date:"2008-07-01"})
 WriteResult({ "nInserted" : 1 })
 > show dbs
 admin (empty)
 knw 0.078GB
 local 0.078GB
//You can see that knw database was created

So, the database “knw” and the collection “projects” were created on first document insertion. Note that a collection can contain many documents with different structure.

> db.projects.insert({client:"Currency Museum", 
                      name:"CMM", 
                      description:"Currency Museum management platform"})
WriteResult({ "nInserted" : 1 })

As you can see, column (field) names are not the same (“client” field does not appear in the first document although they are stored in the same collection).

Note also that a field value can be an integer, a string, a date  (etc) or a full document. (like “soldTo” field)

> db.projects.insert({name:"GEDARC", 
                      description:"EDM and ARCHIVES MGMT platform", 
                      soldTO:{ 
                               name:"Client A", 
                               country:"TN", 
                               licence:1, 
                               renew:true
                             }
                     } )
WriteResult({ "nInserted" : 1 })

The select of MongoDB is called “find”. To find our main project “DALEEL”.

> db.projects.find({name:"DALEEL"})
{ "_id" : ObjectId("547d8321a2fdc9b65ab4805a"), "name" : "DALEEL", "description" : "Directories and VAS plateforms for call centers", "start_date" :
"2008-07-01" }

Now, you can start reading about NoSQL and MongoDB. There is something really humongous about MongoDB: The documentation 😉

One more post, I want you all to check before starting any project based on NoSQL spec, Sarah Mei arguing “Why you should never use MongoDB“.

How a NoSQL DB looks like in real world ? :D

How a NoSQL DB looks like in real world ? 😀

Lucene.Net corrupt indexes

lucene.net

 

 

DotLucene is the dotNet version of Java Lucene API. It is still an open source project with a smaller community. There are many classes that needs to be implemented especially those specific to the Java world. Many open source projects are built on top od DotLucene such as SubText, Lucandra.Net (Lucene and Cassandra) and the awesome RavenDB.
I’ve been using it since 2008 and I want to thank everyone that helped revive this project after two years of hibernation.

When playing with the .Net implementation, I faced the ambiguous exception saying “Read Past EOF”.  Contributors always suspected corrupt indexes. It was not my case: I double checked using the CheckIndex.Status class and it does not reveal any problem. I went through the Lucene.Net DLL code, and focused on the method throwing the error(Refill())


private void Refill()

{
long start = bufferStart + bufferPosition;
long end = start + _bufferSize;
if (end > Length())
// don't read past EOF
end = Length();
int newLength = (int) (end - start);

if (newLength <= 0)// The error is thrown here
throw new System.IO.IOException("read past EOF");

if (buffer == null)
{
NewBuffer(new byte[_bufferSize]); // allocate buffer lazily
SeekInternal(bufferStart);
}
ReadInternal(buffer, 0, newLength);
bufferLength = newLength;
bufferStart = start;
bufferPosition = 0;

To check whether your indexes are corrupt or not, you can use the CheckIndex class as below:

Dim cheker As CheckIndex = New CheckIndex(idxdir)
Dim status As CheckIndex.Status = checker.CheckIndex_Renamed_Method()

The status holds many attributes which can ensure the health of your indexes. Mine shows this:
CheckIndex.Status
So the indexes are healthy, no locks is left there (check locks on file also using Lock Hunter), RW rights are set up correctly, so what was happening?

 

Simply, a coding error ^^’


Dim pth As String = Application.StartupPath + INDEX_PATH + "\" + entityName
Dim idxdir As Lucene.Net.Store.Directory = SimpleFSDirectory.Open(New IO.DirectoryInfo(pth))
Dim crawler As IndexSearcher = New IndexSearcher(idxdir, False)

//and later when iterating through the results (i is the counter)

Dim scrd As ScoreDoc = hts.ScoreDocs(i)

//the error occurs here
Dim doc As Document = crawlr.Doc(scrd.Doc)

Two hours and 25 pages to find out the problem…. the crawler is the index searcher that fetches a given entity (entityName), whereas the  crawlr (without “e” before the “r”) is a global index searcher used somewhere else… That’s all.

A good lesson, do a diff for your team’s updates before starting any bug fix!

gym

SMSC Integration Troobleshooting

This post is just a reminder for me. It may be useful one day for someone else too.
I worked on an application that sends short text messages (SMS) through an SMS gateway based on SMPP3.4 protocol; The work is based on a Java library called SMSLib (v3), which is based on JSMPP itself. The SMS library introduced the “Service” idea on top of the AbstractJSMPP gateway already specified on JSMPP. This “Service” allows many other capabilities such as redundunt and load balanced gateways.
However, the integration with different plateforms was not as easy as I thought because of specific features and configuration on each SMSC.
For instance, I was facing a weird “Session not bound exception” while dealing with Nokia SMSC 8.1. Actually, the server answer was:

Bind Response header (31, 80000002, 00000450, 1)

The server response code is 0x000000450. The first step to solve this issue is to understand this error. The problem is, that this code does not belong to standard SMPP 3.4 protocol standard errors!

vendor specific errors

Vendor specific errors

I gathered the vendor documentation for the client SMSC (Nokia SMSC Center 8.1), and the error mentioned is “Too many users with this login ID”.

Nokia Vendor Documentation

Actually the process of sending messages was a background scheduled task using Quartz. It gathers a queue from an oracle table and tries to send items one by one. When it fails to send and the application gets no more connected to the SMSC, it tries to reconnect. It seems that the SMSC is not handling correctly the disconnection on session unbind event(send failure). So, the number of opened session with the same ID keeps growing until it fails to create a new session. The administrator of the SMSC told me after a while(4 months approximately!), that the maximum number of session to open with same ID is 10 🙂

Another ambiguous error received from another type of SMSC (I don’t know even the vendor of this one) is 0x00000008 (System Error – ESME_RSYSERR) . In this type of errors, only the SMSC logs can help you. I want to thank the guy on the other side who was very helpful to get rid of it quickly. Actually, it was related to numbering format and to mandatory params. This is a specific configuration too. In this case, the SMSC is configured to reject any SMS without a source phone number. So, adding the instruction (setFrom) to the outgoing message solved the issue. In the first SMSC, the parameter was not mandatory, and it was gathered by the SMSC from the client ID.

I also added a custom implementation of JSMPPGatway to handle variable transaction timer; Due to some network latency, the transaction time was exceeding the default value assigned on Jsmpp abstract definition (on session creation); The default value is 2000 ms(=2s).  I managed to get it longer by overriding the constructor of JSMPPGateway to handle the transaction timer parameter.

 

IBM reported process hang for long batch processes

I was looking into WAS logs, while I saw the warning 

WSVR0605W: Thread “WebContainer : 2” (0000005a) has been active for 7305026 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung.

I went deeper in the stacktrace, and I found  that this particular process is a batch function that may work for 24hours without any return (batch cleaning job). Well, Webpshere has a thread monitoring policy that will consider a thread ganging after a predefined interval (600 seconds by default). If the process finishes job after this the threshold interval, WAS will report a false alarm as an apologize and will increase threashold interval if this false alarm occurs many times(100) by 1.5 times. 

hmmm… I don’t see this behavior in log files although the process completed after 5 hours…Moreover, the application reported an error (channel call failure).

A quick way to avoid this is to disable thread hang monitoring by setting com.ibm.websphere.threadmonitor.interval property to zero or less. This may not be a good choice for many applications as it may hide bigger problems.

The safer way is to calculte the maximum of the time taken by any process in your application and set com.ibm.websphere.threadmonitor.threshold to a suitable value. In my application that will never be the case as I’m not the full owner of the database, and I found that sometimes, billing system takes all available accesses to the database for many days.

 

Annestou ….

Image

WAS7 – Exceeded the number of allowable managed connection on thread

While using a managed connection on WAS, I’ve been always facing this warning. I wasn’t aware of its severity, until we run to some data loss. The full Warning message is :

0000008d PoolManager W Exceeded the number of allowable managed connection on thread 0000008d. 
1 managed connections are already being used on this thread. 
Managed connection being used on this thread
MCWrapper id 3a473a47 Managed connection WSRdbManagedConnectionImpl@57ce57ce 
State:STATE_TRAN_WRAPPER_INUSE Thread Id: 0000008d Thread Name: WebContainer : 
12 Used with transaction com.ibm.ws.LocalTransaction.LocalTranCoordImpl@36c536c5;RUNNING;

Solving this problem is as simple as :

  1. Go to Data sources > your Datasource or connection factory > Connection pools > Custom properties
  2. Add the custom property: maxNumberOfMCsAllowableInThread with a higher value (In my case 30 was enough)

You do not even need to restart the server as explained here.