Apache Sqoop Pros and Cons

Apache is very helpful in extracting big data set with minimal time .It can be integrated and implemented with many of similar application where big data is involved with frequency use Review collected by and hosted on G2.com.

Sometimes query takes more time in execution when many join or left outer join or other join involved with extra filter in where condition . The failure during partial import happened in long query Review collected by and hosted on G2.com.

The simplicity with which the tool can be used at the get with minimal setup in a distributed environment and the short learning curve. Review collected by and hosted on G2.com.

The logging seemed to be something I personally struggled with identifying data anomalies when came to data movement in my use cases. Review collected by and hosted on G2.com.

Data transfer is in parallel,making it fast and cost effective. Review collected by and hosted on G2.com.

The failure during partial import and export need special handling. Review collected by and hosted on G2.com.

Incremental imports are most useful in sqoop Review collected by and hosted on G2.com.

sometimes when the same database is used for other business applications and my queries involve multiple joins, performance is impacted Review collected by and hosted on G2.com.

Best thing about is it excutes the data transfer parallel.It allows to transfer the data from variety of structured databases. It has huge support community. Review collected by and hosted on G2.com.

Under the hood it uses MapReduce which takes time for even small data transfer. Implementing Change data capture and incremental loads is quite complex.It cannot be paused and resumed. Review collected by and hosted on G2.com.

The best thing about apache sqoop is it provides easy configuration for getting the data in real time from source system Review collected by and hosted on G2.com.

The thing which i disliked about apache sqoop is once the pipleline is broked then it is tough to recover lost messages Review collected by and hosted on G2.com.

The usage is very simple. It's very user friendly. We need not write lot of lines of code to get the data from db or write back to db Review collected by and hosted on G2.com.

There is nothing I can see as of now. If we get supporfor swooping nosql dbs that would be great Review collected by and hosted on G2.com.

The simplicity and effectiveness of the application wins my heart. Also, user-centric design is awesome. I am using Sqoop to import data from external datastores into Hadoop Distributed File System or related Hadoop eco-systems like Hive and HBase. The best benefit is how easy it is to use and how fast it is. Sqoop can easily integrate with Hadoop and dump structured data from relational databases on HDFS, complimenting the power of Hadoop. This is why Big Data and Hadoop certification mandates a sound knowledge of Apache Sqoop and Flume. Review collected by and hosted on G2.com.

I did not really find anything that I didn't like but if in future I did, I would love to share it. There are few issues using sqoop which buged me initially but are easy to handle:

Sqoop Connector:

Issue:

Use of incorrect connector for the Database to be connected while doing sqoop export or sqoop import.

Missing driver or use of correct driver name of the respective jdbc class for sqoop command.

Missing connection-manager name in some cases of sqoop command.

Incorrect approach of giving password or username of the Database to be connected.

Format of the data stored in HDFS/Hive Tables can create issues. There are few formats such as ORC files which wont allow direct data transfer using sqoop.

Non-matching or incorrect names of columns of source and destination tables where HCatalog is used in sqoop command can show successful sqoop job without the data being transferred. Review collected by and hosted on G2.com.

Replication of Relational DB onto HDFS for MR jobs Review collected by and hosted on G2.com.

The data had to be re-imported every time the data was changed Review collected by and hosted on G2.com.