Using log4j to get ibatis and SQL logs

# Global logging configuration
log4j.rootLogger=ERROR, stdout

#log4j.logger.com.ibatis=DEBUG

# shows SQL of prepared statements
#log4j.logger.java.sql.Connection=DEBUG

# shows parameters inserted into prepared statements
#log4j.logger.java.sql.PreparedStatement=DEBUG

# shows query results
#log4j.logger.java.sql.ResultSet=DEBUG

#log4j.logger.java.sql.Statement=DEBUG

# Console output
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] - %m%n

Anatomy of Spam Business

This paper details the way Spam Business model works.

Paper presents the work in great detail and it is astonishing to know that Spammers have a ecosystem in which they thrive. They setup their stores through affiliate program and host them in bullet-proof hosting servers which are managed by companies which do not abide by request to take down the site.

Elastic Search - an interesting search solution

ElasticSearch is a open source RESTful Search Server, built on top of Lucene Library.
It boasts of following features

• JSON over HTTP


• Free Search Schema


• Near Realtime search


• Easy Distributed Index and Search


• Multi-Tenancy


• Ready for Cloud - very easy for setting it up in Amazon Cloud


• JAVA API Support


• Support for Facets


• It uses Write Behind Queue to store Index updates. It makes use of TransactionLogs to keep track of Index updates.


• Reads can be done on Shard Replicas.

However, it does not support XML.

On the surface, it appears that this product is ready for Web 2.0 world and is ideal for cloud deployment.

Its feature set is not as rich as Apache Solr but it does have decent support for Facets which is hot nowadays. It has very good Data Visualization support which makes it ideal for Monitoring Tools.

How does it compare with Solr?

• Solr is richer in feature set, w.r.t analyzers and facets.


• Solr's distributed setup is not ideal and looks awkward. ElasticSearch's design seems to be robust.


• Solr has been there for much longer and has matured community behind it.


• ElasticSearch is so far only one committer's work.


• ElasticSearch scores over Solr in terms of Cloud Readyness.


• XML support is misisng in ElasticSearch which is not a big deal as JSON is standard for Web 2.0 world.
  

You can get more info from these slides.

When do you use ES?

• Big Index or Realtime Search is needed


• or, there are many indexes


• or, have a multi-tenancy requirement ( Solr core is okay)

When you should not use ES?

• If team is comfortable with Solr then stick to it


• justifying ES in a large corp would be difficult

More info can be obtained from here.

Logstash: A Free/Open Source alternative to Splunk

Today I came across a wonderful presentation on logstash, a open source log archiver and analyzer which makes use of ElasticSearch to index and search log data.

What makes it interesting is, it has very good support for collecting events from different sources such as log files, sys logs, sockets as well as MQ. It will let you apply different filters and stores its index in ElasticSearch.

Use of elasticsearch is interesting as it uses JSON to index/read data and provides an easy way to search and visualize log data. ElasticSeach can scale better than Solr and is ready for Cloud

This is a compelling package and offers a credible alternative to Splunk.

logstash project url is this.

How to log httpclient using log4j and Java Util logging

there are times when httpclient's trace needs to be logged

we can use a log4j file like this

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p [%c] %m%n

log4j.logger.org.apache.http=DEBUG
log4j.logger.org.apache.http.wire=ERROR


You can pass the path of this config file at command line like this
-Dlog4j.configuration=C:\myworkspaces\Client\src\log4j.properties

if you are using Java util logging then you can use this
-Djava.util.logging.config.file=C:\myworkspaces\Client\src\logging.properties

java util log file needs to be like this

.level = FINEST

handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
java.util.logging.ConsoleHandler.level = ALL

org.apache.http.level = FINEST
org.apache.http.wire.level = SEVERE

We are in the price rise age

Days of cheap stuff seem to be gone. It looks like price of commodity and everything else is about to go up or is already inching up.

As per, Jeremy Grantham times are changing and we are in for rude shock.

Here is the summary of Grantham’s thoughtful newsletter: (lifted from here)

• Until about 1800, our species had no safety margin and lived, like other animals, up to the limit of the food supply, ebbing and fl owing in population.

• From about 1800 on the use of hydrocarbons allowed for an explosion in energy use, in food supply, and, through the creation of surpluses, a dramatic increase in wealth and scientific progress.

• Since 1800, the population has surged from 800 million to 7 billion, on its way to an estimated 8 billion, at minimum.

• The rise in population, the ten-fold increase in wealth in developed countries, and the current explosive growth in developing countries have eaten rapidly into our finite resources of hydrocarbons and metals, fertilizer, available land, and water.

• Now, despite a massive increase in fertilizer use, the growth in crop yields per acre has declined from 3.5% in the 1960s to 1.2% today. There is little productive new land to bring on and, as people get richer, they eat more grain-intensive meat. Because the population continues to grow at over 1%, there is little safety margin.

• The problems of compounding growth in the face of finite resources are not easily understood by optimistic, short-term-oriented, and relatively innumerate humans (especially the political variety).

• The fact is that no compound growth is sustainable. If we maintain our desperate focus on growth, we will run out of everything and crash. We must substitute qualitative growth for quantitative growth.

• But Mrs. Market is helping, and right now she is sending us the Mother of all price signals. The prices of all important commodities except oil declined for 100 years until 2002, by an average of 70%. From 2002 until now, this entire decline was erased by a bigger price surge than occurred during World War II.

• Statistically, most commodities are now so far away from their former downward trend that it makes it very probable that the old trend has changed – that there is in fact a Paradigm Shift – perhaps the most important economic event since the Industrial Revolution.

• Climate change is associated with weather instability, but the last year was exceptionally bad. Near term it will surely get less bad.

• Excellent long-term investment opportunities in resources and resource efficiency are compromised by the high chance of an improvement in weather next year and by the possibility that China may stumble.

• From now on, price pressure and shortages of resources will be a permanent feature of our lives. This will increasingly slow down the growth rate of the developed and developing world and put a severe burden on poor countries.

• We all need to develop serious resource plans, particularly energy policies. There is little time to waste.

Urbanization of India - the road ahead

Recent controversy about Lavasa Project shows that India has to evolve a lot for much needed urban renewal of country.

As India prospers and agricultural sector languishes, farmers and workers are moving to cities to earn their livelihood. This is creating pressure on cities and we need to build new cities. Lavasa can be a good model but for our rotten system.

However, we have to keep working on our system so that new cities can be built.

you can read about lavasa here

Using Like in ibatis

when we use like operator, we normally need to do something like this

select * from t where t.name like 'saroj%'

we pass % at the end or beginning or both

if we had to do the same thing in IBatis using dynamic inputs then we can do following
select * from t where name like #name#||'%'

if we have a list of dynamic inputs then following would work

select * from t
where

name like #name[]#||'%'


this would generate following SQL:

select * from t where name like 'saroj%' or name like 'name1%' or name like 'name2%'

How to develop critical thinking at early age

Critical Thinking is a must for a fulfilling life.

Question is: How do we build it? Well, this article gives good tip for building it in early childhood.

Key ideas are

1) Reason with your kid.

2) Classify ideas and share them with kid.

3) Build relationship between objects

4) Ask why, what if and why not?

5) Explore alternative ways

Eclipse Galileo and Helios won't update behind proxy

If you are trying to update Eclipse Galileo (3.5) and Helios (3.6) behind Proxy, you might run into problems like "Site is not found".

So, obvious course of action would be to check Proxy Configuration. Now, you change the settings and ensure that it has right authorization details.

Even after these steps, eclipse would fail to update and error message remains same.

It sounds baffling and it sure is.

After doing some googling for it, I found that Eclipse 3.5 onwards, FileTransferAPI has been changed. In ECF 3.0/Eclipse 3.5 the primary provider is based upon Apache httpclient 3.1. This was introduced in the ECF 3.0/Eclipse 3.5 cycle because the previous provider that was based upon the JRE URLConnection implementation proved insufficiently reliable (i.e. see bug 166179).

Unfortunately, the Apache httpclient implementation, although more robust than the URLConnection-based provider, does not support NTLMv2 proxies directly (for an explanation of why, see here).

For NTLMv2 Proxies, that require username and password for access the workaround is to

1. Disable the ECF httpclient provider.
2. Provide the NTLMv2 proxy authentication info (proxyhost, domain, username, and password)

In ECF 3.0/Galileo both can be done via system properties provided to Eclipse on startup. Here is an example using 'myproxy', 'mydomain', 'myusername', and 'mypassword':

Following settings can be put in eclipse.ini so that Eclipse does not use HTTPClient

-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient
-Dhttp.proxyPort=8080
-Dhttp.proxyHost=myproxy
-Dhttp.proxyUser=mydomain\myusername
-Dhttp.proxyPassword=mypassword
-Dhttp.nonProxyHosts=localhost|127.0.0.1


once you do this, you should be able to update Eclipse.

References:

1. ECF Filetransfer Support for NTLMv2 Proxy
2. Eclipse Bug about this problem
   

Setting up Apache Solr in Eclipse with Tomcat

Prerequisites for this exercise:
• Apache Solr 1.4.1 ( or some other release)
• Eclipse WTP
• Apache Tomcat 6.x

Steps are:

1. Extract Solr-1.4.1 release into C:\ Drive. For my example, directory is C:\apache-solr-1.4.1

2. Launch Eclipse and create a workspace, say solr-workspace

3. Import apache-solr-1.4.1.war file in your workspace by clicking on FIle -> Import,
Select Web --> War as the Import Source.
WAR File: C:\apache-solr-1.4.1\dist\apache-solr-1.4.1.war
Web Project: apache-solr-1.4.1
Click on Finish.






4. Create a folder called solr in the project and create another folder called conf inside solr as shown below:





5. Copy files from C:\apache-solr-1.4.1\client\ruby\solr-ruby\solr\conf into /solr/conf folder.
You can define your schema in schema.xml






6. Create a Tomcat Server using server wizard. Defaults would work fine.




7. Add project apache-solr-1.4.1 to your server as shown below


8. Edit server.xml to let tomcat know where are solr config files are:

<Context docBase="apache-solr-1.4.1" path="/apache-solr-1.4.1"
reloadable="true" source="org.eclipse.jst.j2ee.server:apache-solr-1.4.1">

<Environment name="solr/home" type="java.lang.String"
value="C:\myworkspaces\solr-workspace\apache-solr-1.4.1\solr" override="true"/>

</Context>

9. Start Server and you should see following

INFO: Using JNDI solr.home: C:\myworkspaces\solr-workspace\apache-solr-1.4.1\solr
INFO: Solr home set to 'C:\myworkspaces\solr-workspace\apache-solr-1.4.1\solr\'

Troubleshooting Tips

If your installation is showing errors that solr home is not found then do following
• ensure that server.xml is having right entry
• solr directory exists and has conf as child folder and all the files are there
• clean tomcat-server and clean tomcat work directory

Computer says no

Good read about how messy the computer systems are, at big banks.

As a result banks tend to operate lots of different databases producing conflicting numbers. “The reality was you could never be certain that anything was correct,” says a former executive at Royal Bank of Scotland. Reported numbers for the bank’s exposure were regularly billions of dollars adrift of reality, he reports; finding the source of the error was hard.