Sunday, June 18, 2017

setup local cassandra repository coexist with cassandra upstream

For the past several years, apache cassandra has always been in my working realm. From start of cassandra 0.8 until cassandra 1.2 to beyond cassandra 3.9 (as of this time), I have been using it by modelling data in cassandra, inserting, retrieving, administrating and maintenance of production cluster.

So to take a step further, I thought of going into cassandra development. In this article, I will describe how I got source from cassandra official git repository and setup my own repository in github , so to coexist both of them.

Previously I have done git clone http://git-wip-us.apache.org/repos/asf/cassandra.git

 user@localhost:~/cassandra-trunk$ git remote -v  
 origin     http://git-wip-us.apache.org/repos/asf/cassandra.git (fetch)  
 origin     http://git-wip-us.apache.org/repos/asf/cassandra.git (push)  

So pretty much usual. Now, let's change to the follow
* origin point to github repository
* upstream point to github repository

First, let's remove the remote origin and then add my repository in github. Of cause, create an empty repository in github first before you continue following.

 user@localhost:~/cassandra-trunk$ git remote remove origin  
 user@localhost:~/cassandra-trunk$ git remote add origin https://github.com/jasonwee/cassandra.git  

Now, let's check the remote origin.

 user@localhost:~/cassandra-trunk$ git remote -v  
 origin     https://github.com/jasonwee/cassandra.git (fetch)  
 origin     https://github.com/jasonwee/cassandra.git (push)  

okay, everything is on track and expected. Now let's add upstream to the cassandra git repository.

 user@localhost:~/cassandra-trunk$ git remote add upstream http://git-wip-us.apache.org/repos/asf/cassandra.git  

and then we check again.

 user@localhost:~/workspace/StudyCassandra/cassandra-trunk$ git remote -v  
 origin     https://github.com/jasonwee/cassandra.git (fetch)  
 origin     https://github.com/jasonwee/cassandra.git (push)  
 upstream     http://git-wip-us.apache.org/repos/asf/cassandra.git (fetch)  
 upstream     http://git-wip-us.apache.org/repos/asf/cassandra.git (push)  

OK! everything is good to go. okay.. now that we have two remote repositories, so how should we continue to work further? Now, when we pull, we have to first specify where to pull from and what branch to pull. In the following example, we pull from upstream on the trunk (master) branch.

 user@localhost:~/cassandra-trunk$ git pull upstream trunk  
 From http://git-wip-us.apache.org/repos/asf/cassandra  
  * branch      trunk   -> FETCH_HEAD  
 Already up-to-date.  

beautiful, now we can pull from upstream and into our working repository. Let's push our repository into github now.

 user@localhost:~/cassandra-trunk$ git push -u origin trunk  
 Counting objects: 266270, done.  
 Delta compression using up to 8 threads.  
 Compressing objects: 100% (42703/42703), done.  
 Writing objects: 100% (266270/266270), 136.16 MiB | 547.00 KiB/s, done.  
 Total 266270 (delta 160130), reused 265316 (delta 159364)  
 remote: Resolving deltas: 100% (160130/160130), done.  
 To https://github.com/jasonwee/cassandra.git  
  * [new branch]   trunk -> trunk  
 Branch trunk set up to track remote branch trunk from origin.  

okay, that's it, one more step ahead.

Saturday, June 17, 2017

how to debug remote cassandra with eclipse

This tutorial is written such a way you install cassandra using debian package and want to debug remote apache cassandra instance with another workstation with eclipse install. With that said, let's start.

* uncomment the following lines in this cassandra environment file /etc/cassandra/cassandra-env.sh . The cassandra environment for debian is located at the path specified.

 # JVM_OPTS="$JVM_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1414"  

* restart apache cassandra instance and verify the above jvm options are shown in ps output.

 $ ps aux | grep cassandra | grep --color 1414  
 cassand+ 26718 44.8 29.8 1556048 1232024 ?   SLl 19:19  2:28 /usr/lib/jvm/jdk1.8.0_45//bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:+CMSClassUnloadingEnabled -Xms1024M -Xmx1024M -Xmn200M -ea -Xss256k -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:StringTableSize=1000003 -XX:+UseTLAB -XX:+ResizeTLAB -XX:+PerfDisableSharedMem -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+HeapDumpOnOutOfMemoryError -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1414 -Djava.net.preferIPv4Stack=false -Dcassandra.jmx.local.port=7199 -XX:+DisableExplicitGC -Djava.library.path=/usr/share/cassandra/lib/sigar-bin -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=/var/lib/cassandra -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp /etc/cassandra:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.4.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.0-beta1-bb1bce4-SNAPSHOT-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jgrapht-core-0.9.1.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/ohc-core-0.4.2.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.2.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.0.jar:/usr/share/cassandra/lib/reporter-config3-3.0.0.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-3.0.0.jar:/usr/share/cassandra/apache-cassandra-thrift-3.0.0.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar: -XX:HeapDumpPath=/var/lib/cassandra/java_1473247174.hprof -XX:ErrorFile=/var/lib/cassandra/hs_err_1473247174.log org.apache.cassandra.service.CassandraDaemon# JVM_OPTS="$JVM_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1414"  

* okay looks good, now let's move on to the eclipse. See the screenshot below, connect to it. To get to the debu configuration window, click on 'Run' in the menu, 'Debug Configurations...' on the pop up window, right click 'Remote Java Application' and click new.



* check in the machine that run cassandra instance.

 user@workstation:~$ sudo netstat -tupan | grep 1414  
 tcp    0   0 10.0.0.2:1414    10.0.0.3:60122   ESTABLISHED 26718/java   

That's it! When you are done, remember to terminate the debug process.


Friday, June 16, 2017

Journey on cassandra development part1

To go into cassandra development has always been part of my wish during spare time but due to hectic work and life, this wish never really get kick off.... Until now. I found a very fine documentation and just start immediately once the spare time is available. In this article, I will describe my journey on developing in cassandra.

Let's clone the cassandra repository using git. Wait a while on my slow line at 436KBps with 136MB.

 user@localhost:~/$ git clone http://git-wip-us.apache.org/repos/asf/cassandra.git cassandra-trunk  
 Cloning into 'cassandra-trunk'...  
 remote: Counting objects: 273406, done.  
 remote: Compressing objects: 100% (45601/45601), done.  
 remote: Total 273406 (delta 164011), reused 269444 (delta 161446)  
 Receiving objects: 100% (273406/273406), 136.60 MiB | 436.00 KiB/s, done.  
 Resolving deltas: 100% (164011/164011), done.  
 Checking connectivity... done.  
   

Once repository was cloned, let's check out the directory.

 user@localhost:~/$ cd cassandra-trunk/  
 user@localhost:~/cassandra-trunk$ ls  
 total 596K  
 -rw-r--r-- 1 user user 3.5K Sep 6 21:28 README.asc  
 -rw-r--r-- 1 user user 2.8K Sep 6 21:28 NOTICE.txt  
 -rw-r--r-- 1 user user 99K Sep 6 21:28 NEWS.txt  
 -rw-r--r-- 1 user user 12K Sep 6 21:28 LICENSE.txt  
 -rw-r--r-- 1 user user 1.3K Sep 6 21:28 CONTRIBUTING.md  
 -rw-r--r-- 1 user user 320K Sep 6 21:28 CHANGES.txt  
 drwxr-xr-x 3 user user 4.0K Sep 6 21:28 conf  
 -rw-r--r-- 1 user user 91K Sep 6 21:28 build.xml  
 -rw-r--r-- 1 user user 516 Sep 6 21:28 build.properties.default  
 drwxr-xr-x 2 user user 4.0K Sep 6 21:28 bin  
 drwxr-xr-x 4 user user 4.0K Sep 6 21:28 doc  
 drwxr-xr-x 3 user user 4.0K Sep 6 21:28 debian  
 drwxr-xr-x 3 user user 4.0K Sep 6 21:28 interface  
 drwxr-xr-x 3 user user 4.0K Sep 6 21:28 ide  
 drwxr-xr-x 4 user user 4.0K Sep 6 21:28 examples  
 -rw-r--r-- 1 user user 5.8K Sep 6 21:28 eclipse_compiler.properties  
 drwxr-xr-x 3 user user 4.0K Sep 6 21:28 pylib  
 drwxr-xr-x 5 user user 4.0K Sep 6 21:28 lib  
 drwxr-xr-x 6 user user 4.0K Sep 6 21:28 src  
 drwxr-xr-x 9 user user 4.0K Sep 6 21:28 test  
 drwxr-xr-x 4 user user 4.0K Sep 6 21:28 tools  
 user@localhost:~/cassandra-trunk$ git branch  
 * trunk  
 user@localhost:~/cassandra-trunk$ git branch -a  
 * trunk  
  remotes/origin/HEAD -> origin/trunk  
  remotes/origin/cassandra-1.0  
  remotes/origin/cassandra-1.1  
  remotes/origin/cassandra-1.2  
  remotes/origin/cassandra-2.0  
  remotes/origin/cassandra-2.1  
  remotes/origin/cassandra-2.2  
  remotes/origin/cassandra-3.0  
  remotes/origin/cassandra-3.9  
  remotes/origin/trunk  

okay, looks almost the same sets of files when you downloaded the binary tarball package except this is much more. But that is expected because we are  in development environment. As of this moment, we have cassandra up to version 3.9

Before we go further, let's checkout the package required by cassandra.

 user@localhost:~/cassandra-trunk$ ant -version  
 Apache Ant(TM) version 1.9.7 compiled on May 16 2016  
 user@localhost:~/cassandra-trunk$ export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_45/  

So I'm using trunk cassandra , ant version 1.9.7 and oracle jdk version8 update 45. Eclipse I'm a bit behind, I'm still sticking with eclipse luna, I should  upgrade to eclipse neon soon!

Okay, our tools are all checked at this point, let's start ant build.

 user@localhost:~/cassandra-trunk$ ant  
 Buildfile: /home/user/cassandra-trunk/build.xml  
   
 init:  
   [mkdir] Created dir: /home/user/cassandra-trunk/build/classes/main  
   [mkdir] Created dir: /home/user/cassandra-trunk/build/classes/thrift  
   [mkdir] Created dir: /home/user/cassandra-trunk/build/test/lib  
   [mkdir] Created dir: /home/user/cassandra-trunk/build/test/classes  
   [mkdir] Created dir: /home/user/cassandra-trunk/build/test/stress-classes  
   [mkdir] Created dir: /home/user/cassandra-trunk/src/gen-java  
   [mkdir] Created dir: /home/user/cassandra-trunk/build/lib  
   [mkdir] Created dir: /home/user/cassandra-trunk/build/jacoco  
   [mkdir] Created dir: /home/user/cassandra-trunk/build/jacoco/partials  
   
 maven-ant-tasks-localrepo:  
   
 maven-ant-tasks-download:  
    [echo] Downloading Maven ANT Tasks...  
    [get] Getting: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar  
    [get] To: /home/user/cassandra-trunk/build/maven-ant-tasks-2.1.3.jar  
    [copy] Copying 1 file to /home/user/.m2/repository/org/apache/maven/maven-ant-tasks/2.1.3  
   
 maven-ant-tasks-init:  
   
 maven-declare-dependencies:  
   
 maven-ant-tasks-retrieve-build:  
 [artifact:dependencies] Downloading: com/datastax/cassandra/cassandra-driver-core/3.0.1/cassandra-driver-core-3.0.1.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 15K from central  
 [artifact:dependencies] Downloading: com/datastax/cassandra/cassandra-driver-parent/3.0.1/cassandra-driver-parent-3.0.1.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 18K from central  
 [artifact:dependencies] Downloading: io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 1K from central  
 [artifact:dependencies] Downloading: io/dropwizard/metrics/metrics-parent/3.1.2/metrics-parent-3.1.2.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 12K from central  
 [artifact:dependencies] Downloading: org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 3K from central  
 [artifact:dependencies] Downloading: org/slf4j/slf4j-parent/1.7.7/slf4j-parent-1.7.7.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 12K from central  
 [artifact:dependencies] Downloading: org/eclipse/jdt/core/compiler/ecj/4.4.2/ecj-4.4.2.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 2K from central  
 [artifact:dependencies] Downloading: org/caffinitas/ohc/ohc-core/0.4.4/ohc-core-0.4.4.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 11K from central  
 [artifact:dependencies] Downloading: org/caffinitas/ohc/ohc-parent/0.4.4/ohc-parent-0.4.4.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 16K from central  
 [artifact:dependencies] Downloading: org/caffinitas/ohc/ohc-core-j8/0.4.4/ohc-core-j8-0.4.4.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 5K from central  
 [artifact:dependencies] Downloading: org/openjdk/jmh/jmh-core/1.13/jmh-core-1.13.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 10K from central  
 [artifact:dependencies] Downloading: org/openjdk/jmh/jmh-parent/1.13/jmh-parent-1.13.pom from repository java.net at http://download.java.net/maven/2  
 [artifact:dependencies] Unable to locate resource in repository  
 [artifact:dependencies] [INFO] Unable to find resource 'org.openjdk.jmh:jmh-parent:pom:1.13' in repository java.net (http://download.java.net/maven/2)  
 [artifact:dependencies] Downloading: org/openjdk/jmh/jmh-parent/1.13/jmh-parent-1.13.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 6K from central  
 [artifact:dependencies] Downloading: net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 11K from central  
 [artifact:dependencies] Downloading: org/apache/commons/commons-math3/3.2/commons-math3-3.2.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 17K from central  
 [artifact:dependencies] Downloading: org/openjdk/jmh/jmh-generator-annprocess/1.13/jmh-generator-annprocess-1.13.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 4K from central  
 [artifact:dependencies] Downloading: net/ju-n/compile-command-annotations/compile-command-annotations/1.2.0/compile-command-annotations-1.2.0.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 5K from central  
 [artifact:dependencies] Downloading: net/ju-n/net-ju-n-parent/32/net-ju-n-parent-32.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 21K from central  
 [artifact:dependencies] Downloading: org/apache/ant/ant-junit/1.9.4/ant-junit-1.9.4.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 4K from central  
 [artifact:dependencies] Downloading: org/caffinitas/ohc/ohc-core/0.4.4/ohc-core-0.4.4.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Downloading: org/apache/ant/ant-junit/1.9.4/ant-junit-1.9.4.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Downloading: org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Downloading: io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Downloading: net/ju-n/compile-command-annotations/compile-command-annotations/1.2.0/compile-command-annotations-1.2.0.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 132K from central  
 [artifact:dependencies] Transferring 16K from central  
 [artifact:dependencies] Transferring 110K from central  
 [artifact:dependencies] Transferring 1653K from central  
 [artifact:dependencies] Downloading: org/eclipse/jdt/core/compiler/ecj/4.4.2/ecj-4.4.2.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 2256K from central  
 [artifact:dependencies] Downloading: net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 115K from central  
 [artifact:dependencies] Downloading: com/datastax/cassandra/cassandra-driver-core/3.0.1/cassandra-driver-core-3.0.1-shaded.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 61K from central  
 [artifact:dependencies] Transferring 2388K from central  
 [artifact:dependencies] Downloading: org/openjdk/jmh/jmh-core/1.13/jmh-core-1.13.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 454K from central  
 [artifact:dependencies] Downloading: org/caffinitas/ohc/ohc-core-j8/0.4.4/ohc-core-j8-0.4.4.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 5K from central  
 [artifact:dependencies] Downloading: org/openjdk/jmh/jmh-generator-annprocess/1.13/jmh-generator-annprocess-1.13.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 30K from central  
 ^Y[artifact:dependencies] Building ant file: /home/user/cassandra-trunk/build/build-dependencies.xml  
 [artifact:dependencies] Downloading: io/netty/netty-all/4.0.39.Final/netty-all-4.0.39.Final.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 17K from central  
 [artifact:dependencies] Downloading: io/netty/netty-parent/4.0.39.Final/netty-parent-4.0.39.Final.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 44K from central  
 [artifact:dependencies] Downloading: io/netty/netty-all/4.0.39.Final/netty-all-4.0.39.Final.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 2218K from central  
 [artifact:dependencies] Downloading: com/datastax/cassandra/cassandra-driver-core/3.0.1/cassandra-driver-core-3.0.1-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 552K from central  
 [artifact:dependencies] Downloading: io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 52K from central  
 [artifact:dependencies] Downloading: org/slf4j/slf4j-api/1.7.12/slf4j-api-1.7.12-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 50K from central  
 [artifact:dependencies] Downloading: io/netty/netty-all/4.0.39.Final/netty-all-4.0.39.Final-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 1749K from central  
 [artifact:dependencies] Downloading: org/eclipse/jdt/core/compiler/ecj/4.4.2/ecj-4.4.2-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 1724K from central  
 [artifact:dependencies] Downloading: org/caffinitas/ohc/ohc-core/0.4.4/ohc-core-0.4.4-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 83K from central  
 [artifact:dependencies] Downloading: org/openjdk/jmh/jmh-core/1.13/jmh-core-1.13-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 357K from central  
 [artifact:dependencies] Downloading: net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 73K from central  
 [artifact:dependencies] Downloading: org/apache/commons/commons-math3/3.2/commons-math3-3.2-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 1958K from central  
 [artifact:dependencies] Downloading: org/openjdk/jmh/jmh-generator-annprocess/1.13/jmh-generator-annprocess-1.13-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 26K from central  
 [artifact:dependencies] Downloading: net/ju-n/compile-command-annotations/compile-command-annotations/1.2.0/compile-command-annotations-1.2.0-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 15K from central  
 [artifact:dependencies] Downloading: org/apache/ant/ant-junit/1.9.4/ant-junit-1.9.4-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 92K from central  
 [artifact:dependencies] Downloading: org/apache/ant/ant/1.9.4/ant-1.9.4-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 1889K from central  
 [artifact:dependencies] Downloading: org/apache/ant/ant-launcher/1.9.4/ant-launcher-1.9.4-sources.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 19K from central  
 [artifact:dependencies] Building ant file: /home/user/cassandra-trunk/build/build-dependencies-sources.xml  
    [copy] Copying 62 files to /home/user/cassandra-trunk/build/lib/jars  
    [copy] Copying 17 files to /home/user/cassandra-trunk/build/lib/sources  
 [artifact:dependencies] Downloading: org/jacoco/org.jacoco.agent/0.7.5.201505241946/org.jacoco.agent-0.7.5.201505241946.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 3K from central  
 [artifact:dependencies] Downloading: org/jacoco/org.jacoco.build/0.7.5.201505241946/org.jacoco.build-0.7.5.201505241946.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 36K from central  
 [artifact:dependencies] Downloading: org/jacoco/org.jacoco.ant/0.7.5.201505241946/org.jacoco.ant-0.7.5.201505241946.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 2K from central  
 [artifact:dependencies] Downloading: org/jacoco/org.jacoco.core/0.7.5.201505241946/org.jacoco.core-0.7.5.201505241946.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 1K from central  
 [artifact:dependencies] Downloading: org/ow2/asm/asm-debug-all/5.0.1/asm-debug-all-5.0.1.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 2K from central  
 [artifact:dependencies] Downloading: org/ow2/asm/asm-parent/5.0.1/asm-parent-5.0.1.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 5K from central  
 [artifact:dependencies] Downloading: org/jacoco/org.jacoco.report/0.7.5.201505241946/org.jacoco.report-0.7.5.201505241946.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 1K from central  
 [artifact:dependencies] Downloading: org/jboss/byteman/byteman/3.0.3/byteman-3.0.3.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 91K from central  
 [artifact:dependencies] Downloading: org/jboss/byteman/byteman-root/3.0.3/byteman-root-3.0.3.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 18K from central  
 [artifact:dependencies] Downloading: org/jboss/byteman/byteman-submit/3.0.3/byteman-submit-3.0.3.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 2K from central  
 [artifact:dependencies] Downloading: org/jboss/byteman/byteman-bmunit/3.0.3/byteman-bmunit-3.0.3.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 6K from central  
 [artifact:dependencies] Downloading: org/jboss/byteman/byteman-install/3.0.3/byteman-install-3.0.3.pom from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 2K from central  
 [artifact:dependencies] Downloading: org/jacoco/org.jacoco.agent/0.7.5.201505241946/org.jacoco.agent-0.7.5.201505241946.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Downloading: org/ow2/asm/asm-debug-all/5.0.1/asm-debug-all-5.0.1.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Downloading: org/jboss/byteman/byteman/3.0.3/byteman-3.0.3.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 251K from central  
 [artifact:dependencies] Transferring 743K from central  
 [artifact:dependencies] Transferring 371K from central  
 [artifact:dependencies] Downloading: org/jboss/byteman/byteman-submit/3.0.3/byteman-submit-3.0.3.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 14K from central  
 [artifact:dependencies] Downloading: org/jboss/byteman/byteman-bmunit/3.0.3/byteman-bmunit-3.0.3.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Downloading: org/jacoco/org.jacoco.ant/0.7.5.201505241946/org.jacoco.ant-0.7.5.201505241946.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 38K from central  
 [artifact:dependencies] Transferring 37K from central  
 [artifact:dependencies] Downloading: org/jboss/byteman/byteman-install/3.0.3/byteman-install-3.0.3.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 9K from central  
 [artifact:dependencies] Downloading: org/jacoco/org.jacoco.core/0.7.5.201505241946/org.jacoco.core-0.7.5.201505241946.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 130K from central  
 [artifact:dependencies] Downloading: org/jacoco/org.jacoco.report/0.7.5.201505241946/org.jacoco.report-0.7.5.201505241946.jar from repository central at http://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 137K from central  
    [copy] Copying 9 files to /home/user/cassandra-trunk/build/lib/jars  
   [unzip] Expanding: /home/user/cassandra-trunk/build/lib/jars/org.jacoco.agent-0.7.5.201505241946.jar into /home/user/cassandra-trunk/build/lib/jars  
   
 check-gen-cql3-grammar:  
   
 gen-cql3-grammar:  
    [echo] Building Grammar /home/user/cassandra-trunk/src/antlr/Cql.g ...  
   
 generate-cql-html:  
 [artifact:dependencies] Downloading: com/datastax/wikitext/wikitext-core-ant/1.3/wikitext-core-ant-1.3.pom from repository central at https://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 3K from central  
 [artifact:dependencies] Downloading: org/fusesource/wikitext/wikitext-core/1.3/wikitext-core-1.3.pom from repository central at https://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 2K from central  
 [artifact:dependencies] Downloading: org/fusesource/wikitext/wikitext-project/1.3/wikitext-project-1.3.pom from repository central at https://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 4K from central  
 [artifact:dependencies] Downloading: org/fusesource/fusesource-pom/1.3/fusesource-pom-1.3.pom from repository central at https://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 13K from central  
 [artifact:dependencies] Downloading: org/fusesource/wikitext/textile-core/1.3/textile-core-1.3.pom from repository central at https://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 1K from central  
 [artifact:dependencies] Downloading: org/fusesource/wikitext/wikitext-core/1.3/wikitext-core-1.3.jar from repository central at https://repo1.maven.org/maven2  
 [artifact:dependencies] Downloading: com/datastax/wikitext/wikitext-core-ant/1.3/wikitext-core-ant-1.3.jar from repository central at https://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 237K from central  
 [artifact:dependencies] Transferring 32K from central  
 [artifact:dependencies] Downloading: org/fusesource/wikitext/textile-core/1.3/textile-core-1.3.jar from repository central at https://repo1.maven.org/maven2  
 [artifact:dependencies] Transferring 54K from central  
   
 generate-jflex-java:  
   [jflex] Generated: StandardTokenizerImpl.java  
   
 build-project:  
    [echo] apache-cassandra: /home/user/cassandra-trunk/build.xml  
   [javac] Compiling 45 source files to /home/user/cassandra-trunk/build/classes/thrift  
   [javac] Note: /home/user/cassandra-trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java uses or overrides a deprecated API.  
   [javac] Note: Recompile with -Xlint:deprecation for details.  
   [javac] Note: Some input files use unchecked or unsafe operations.  
   [javac] Note: Recompile with -Xlint:unchecked for details.  
   [javac] Compiling 1459 source files to /home/user/cassandra-trunk/build/classes/main  
   [javac] Note: Processing compiler hints annotations  
   [javac] Note: Processing compiler hints annotations  
   [javac] Note: Writing compiler command file at META-INF/hotspot_compiler  
   [javac] Note: Done processing compiler hints annotations  
   [javac] Note: Some input files use or override a deprecated API.  
   [javac] Note: Recompile with -Xlint:deprecation for details.  
   [javac] Note: Some input files use unchecked or unsafe operations.  
   [javac] Note: Recompile with -Xlint:unchecked for details.  
   [javac] Creating empty /home/user/cassandra-trunk/build/classes/main/org/apache/cassandra/hints/package-info.class  
   
 createVersionPropFile:  
   [mkdir] Created dir: /home/user/cassandra-trunk/src/resources/org/apache/cassandra/config  
 [propertyfile] Creating new property file: /home/user/cassandra-trunk/src/resources/org/apache/cassandra/config/version.properties  
    [copy] Copying 18 files to /home/user/cassandra-trunk/build/classes/main  
    [copy] Copying 1 file to /home/user/cassandra-trunk/conf  
   
 build:  
   
 build-test:  
   [javac] Compiling 481 source files to /home/user/cassandra-trunk/build/test/classes  
   [javac] Note: Some input files use or override a deprecated API.  
   [javac] Note: Recompile with -Xlint:deprecation for details.  
   [javac] Note: Some input files use unchecked or unsafe operations.  
   [javac] Note: Recompile with -Xlint:unchecked for details.  
    [copy] Copying 22 files to /home/user/cassandra-trunk/build/test/classes  
   
 stress-build:  
   [mkdir] Created dir: /home/user/cassandra-trunk/build/classes/stress  
   [javac] Compiling 118 source files to /home/user/cassandra-trunk/build/classes/stress  
   [javac] Note: Some input files use or override a deprecated API.  
   [javac] Note: Recompile with -Xlint:deprecation for details.  
   [javac] Note: Some input files use unchecked or unsafe operations.  
   [javac] Note: Recompile with -Xlint:unchecked for details.  
    [copy] Copying 1 file to /home/user/cassandra-trunk/build/classes/stress  
   
 write-poms:  
   
 init:  
   
 maven-ant-tasks-localrepo:  
   
 maven-ant-tasks-download:  
   
 maven-ant-tasks-init:  
   
 maven-declare-dependencies:  
   
 _write-poms:  
   
 jar:  
    [copy] Copying 1 file to /home/user/cassandra-trunk/build/classes/main/META-INF  
    [copy] Copying 1 file to /home/user/cassandra-trunk/build/classes/thrift/META-INF  
    [copy] Copying 1 file to /home/user/cassandra-trunk/build/classes/main/META-INF  
    [copy] Copying 1 file to /home/user/cassandra-trunk/build/classes/thrift/META-INF  
    [jar] Building jar: /home/user/cassandra-trunk/build/apache-cassandra-thrift-3.10-SNAPSHOT.jar  
    [jar] Building jar: /home/user/cassandra-trunk/build/apache-cassandra-3.10-SNAPSHOT.jar  
   [mkdir] Created dir: /home/user/cassandra-trunk/build/classes/stress/META-INF  
   [mkdir] Created dir: /home/user/cassandra-trunk/build/tools/lib  
    [jar] Building jar: /home/user/cassandra-trunk/build/tools/lib/stress.jar  
   
 BUILD SUCCESSFUL  
 Total time: 4 minutes 45 secondsuser@localhost:~/cassandra-trunk$ ant -version  
 Apache Ant(TM) version 1.9.7 compiled on May 16 2016  
 user@localhost:~/cassandra-trunk$ export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_45/  

Wow, build is success! Next launch your eclipse and create cassandra eclipse development descriptors.

 user@localhost:~/cassandra-trunk$ ant generate-eclipse-files  
 Buildfile: /home/user/cassandra-trunk/build.xml  
   
 init:  
   
 maven-ant-tasks-localrepo:  
   
 maven-ant-tasks-download:  
   
 maven-ant-tasks-init:  
   
 maven-declare-dependencies:  
   
 maven-ant-tasks-retrieve-build:  
   
 init-dependencies:  
    [echo] Loading dependency paths from file: /home/user/cassandra-trunk/build/build-dependencies.xml  
   
 init-dependencies:  
    [echo] Loading dependency paths from file: /home/user/cassandra-trunk/build/build-dependencies-sources.xml  
   [unzip] Expanding: /home/user/cassandra-trunk/build/lib/jars/org.jacoco.agent-0.7.5.201505241946.jar into /home/user/cassandra-trunk/build/lib/jars  
   
 check-gen-cql3-grammar:  
   
 gen-cql3-grammar:  
   
 generate-cql-html:  
   
 generate-jflex-java:  
   
 build-project:  
    [echo] apache-cassandra: /home/user/cassandra-trunk/build.xml  
   
 createVersionPropFile:  
 [propertyfile] Updating property file: /home/user/cassandra-trunk/src/resources/org/apache/cassandra/config/version.properties  
    [copy] Copying 1 file to /home/user/cassandra-trunk/build/classes/main  
   
 build:  
   
 build-test:  
   
 generate-eclipse-files:  
   [mkdir] Created dir: /home/user/cassandra-trunk/.settings  
   
 BUILD SUCCESSFUL  
 Total time: 1 second  

Everything is good and assuming eclipse is also started, now let's import the eclipse settings. Using this sequence,'File' ->  'Import...' -> 'Existing Projects into Workspace'. Click, next and in the 'Select root directory:' textbox , browse to the git repository cloned earlier and select it. Leave the remaining as is and click Finish.

If you get error, fix it :D for me, the default java configured for eclipse is 7 and hence, code compile is java 7, so I have to change that and move java lib to the top of the project. Particulary you can move to java lib in 'Order and Export' in project's properties.



There are two class file which give compile errors, which I have to commented out the code.


since both classes are unit test, I don't actually bothered.

Okay, we take a pause here, in the next article, I will explain testing in cassandra. Stay tune!


Sunday, June 4, 2017

First learning into pycuda part 2

So I continue where I left off and explore further on the pycuda library. This is a good tutorial to explain further what the code does. You can read it more here .

 user@localhost:~$ optirun python3 test_cumath.py   
 Traceback (most recent call last):  
  File "test_cumath.py", line 245, in <module>  
   from py.test.cmdline import main  
 ImportError: No module named 'py'  

module py is actually pytest. So install away

 user@localhost:~$ sudo apt-get install python3-pytest  
 Reading package lists... Done  
 Building dependency tree      
 Reading state information... Done  
 The following packages were automatically installed and are no longer required:  
  libgl1-nvidia-glx:i386 libgl1-nvidia-glx-i386:i386 libllvm3.5v5 libnvidia-glcore:i386 linux-image-4.1.0-2-amd64 linux-image-4.2.0-1-amd64 linux-source-4.3 python3-ecdsa syslinux  
  unetbootin-translations  
 Use 'sudo apt autoremove' to remove them.  
 The following additional packages will be installed:  
  python3-py  
 The following NEW packages will be installed:  
  python3-py python3-pytest  
 0 upgraded, 2 newly installed, 0 to remove and 508 not upgraded.  
 Need to get 249 kB of archives.  
 After this operation, 962 kB of additional disk space will be used.  
 Do you want to continue? [Y/n] Y  
 Get:1 http://ftp.us.debian.org/debian testing/main amd64 python3-py all 1.4.31-1 [81.9 kB]  
 Get:2 http://ftp.us.debian.org/debian testing/main amd64 python3-pytest all 2.9.2-3 [167 kB]                                                   
 Fetched 249 kB in 6s (36.0 kB/s)                                                                                 
 Selecting previously unselected package python3-py.  
 (Reading database ... 285403 files and directories currently installed.)  
 Preparing to unpack .../python3-py_1.4.31-1_all.deb ...  
 Unpacking python3-py (1.4.31-1) ...  
 Selecting previously unselected package python3-pytest.  
 Preparing to unpack .../python3-pytest_2.9.2-3_all.deb ...  
 Unpacking python3-pytest (2.9.2-3) ...  
 Processing triggers for man-db (2.7.5-1) ...  
 Setting up python3-py (1.4.31-1) ...  
 Setting up python3-pytest (2.9.2-3) ...  

 user@localhost:~$ optirun python3 -mpytest test_cumath.py   
 ==================================================================================== test session starts =====================================================================================  
 platform linux -- Python 3.5.2+, pytest-2.9.2, py-1.4.31, pluggy-0.3.1  
 rootdir: /home/user/, inifile:   
 collected 28 items   
   
 test_cumath.py ............................  
   
 ================================================================================ 28 passed in 106.94 seconds =================================================================================  

First run is very slow, I have no idea why. Then I run tests again.

 user@localhost:~$ optirun python3 -s -mpytest test_cumath.py   
 ==================================================================================== test session starts =====================================================================================  
 platform linux -- Python 3.5.2+, pytest-2.9.2, py-1.4.31, pluggy-0.3.1  
 rootdir: /home/user/, inifile:   
 collected 28 items   
   
 test_cumath.py ............................  
   
 ================================================================================= 28 passed in 8.43 seconds ==================================================================================  
 user@localhost:~/oss/asus-rt-n14uhp-mrtg/src/pycuda$ optirun python3 -s -mpytest test_cumath.py   
 ==================================================================================== test session starts =====================================================================================  
 platform linux -- Python 3.5.2+, pytest-2.9.2, py-1.4.31, pluggy-0.3.1  
 rootdir: /home/user/, inifile:   
 collected 28 items   
   
 test_cumath.py ............................  
   
 ================================================================================= 28 passed in 8.44 seconds ==================================================================================  

tests average at 8.5seconds, pretty good.  Then I add some code to print the time of the cpu and gpu, looks to me the maths test, cpu perform much better.

 name="<class 'numpy.float32'>" cpu time="0.00000230599835049360990524291992187500000000000000" gpu time="0.00008139800047501921653747558593750000000000000000"  
 name="<class 'numpy.float64'>" cpu time="0.00000265298876911401748657226562500000000000000000" gpu time="0.00008218799484893679618835449218750000000000000000"  
 name="<class 'numpy.float32'>" cpu time="0.00000343500869348645210266113281250000000000000000" gpu time="0.00007703299343120306730270385742187500000000000000"  
 name="<class 'numpy.float64'>" cpu time="0.00000536799780093133449554443359375000000000000000" gpu time="0.00008030000026337802410125732421875000000000000000"  
 name="<class 'numpy.float32'>" cpu time="0.00001179199898615479469299316406250000000000000000" gpu time="0.00007762899622321128845214843750000000000000000000"  
 name="<class 'numpy.float64'>" cpu time="0.00002014498750213533639907836914062500000000000000" gpu time="0.00010288099292665719985961914062500000000000000000"  
 name="<class 'numpy.float32'>" cpu time="0.00001173198688775300979614257812500000000000000000" gpu time="0.00007860400364734232425689697265625000000000000000"  
 name="<class 'numpy.float64'>" cpu time="0.00001884899393189698457717895507812500000000000000" gpu time="0.00007704700692556798458099365234375000000000000000"  
 name="<class 'numpy.float32'>" cpu time="0.00007372199615929275751113891601562500000000000000" gpu time="0.00008238101145252585411071777343750000000000000000"  
 name="<class 'numpy.float64'>" cpu time="0.00014613500388804823160171508789062500000000000000" gpu time="0.00016600399976596236228942871093750000000000000000"  

Have fun and you can get the code from my repository. 

Saturday, June 3, 2017

First learning into pycuda

With last blog which I fail to get a sample working, today I thought of giving pycuda a try. So what is pycuda?

PyCUDA gives you easy, Pythonic access to Nvidia‘s CUDA parallel computation API. 

With that said, I'm gonna give the sample code a try. Let's install python3 pycuda module.

 user@localhost:~/Downloads$ sudo apt-get install python3-pycuda   
 Reading package lists... Done  
 Building dependency tree      
 Reading state information... Done  
 The following packages were automatically installed and are no longer required:  
  libgl1-nvidia-glx:i386 libgl1-nvidia-glx-i386:i386 libllvm3.5v5 libnvidia-glcore:i386 linux-image-4.1.0-2-amd64 linux-image-4.2.0-1-amd64 linux-source-4.3 python3-ecdsa syslinux  
  unetbootin-translations  
 Use 'sudo apt autoremove' to remove them.  
 The following additional packages will be installed:  
  fonts-mathjax libboost-python1.61.0 libboost-system1.61.0 libboost-thread1.61.0 libjs-mathjax python-pycuda-doc python3-appdirs python3-decorator python3-pytools  
 Suggested packages:  
  fonts-mathjax-extras fonts-stix libjs-mathjax-doc python-pycuda python3-pytest python3-opengl python3-pycuda-dbg  
 The following NEW packages will be installed:  
  fonts-mathjax libboost-python1.61.0 libboost-system1.61.0 libboost-thread1.61.0 libjs-mathjax python-pycuda-doc python3-appdirs python3-decorator python3-pycuda python3-pytools  
 0 upgraded, 10 newly installed, 0 to remove and 508 not upgraded.  
 Need to get 7,150 kB of archives.  
 After this operation, 47.8 MB of additional disk space will be used.  
 Do you want to continue? [Y/n] Y  
 Get:1 http://ftp.us.debian.org/debian testing/main amd64 fonts-mathjax all 2.6.1-1 [959 kB]  
 Get:2 http://ftp.us.debian.org/debian testing/main amd64 libboost-python1.61.0 amd64 1.61.0+dfsg-2.1 [137 kB]                                          
 Get:3 http://ftp.us.debian.org/debian testing/main amd64 libboost-system1.61.0 amd64 1.61.0+dfsg-2.1 [32.1 kB]                                          
 Get:4 http://ftp.us.debian.org/debian testing/main amd64 libboost-thread1.61.0 amd64 1.61.0+dfsg-2.1 [71.2 kB]                                          
 Get:5 http://ftp.us.debian.org/debian testing/main amd64 libjs-mathjax all 2.6.1-1 [5,473 kB]                                                  
 Get:6 http://ftp.us.debian.org/debian testing/contrib amd64 python-pycuda-doc all 2016.1-1 [122 kB]                                               
 Get:7 http://ftp.us.debian.org/debian testing/main amd64 python3-appdirs all 1.4.0-2 [11.1 kB]                                                  
 Get:8 http://ftp.us.debian.org/debian testing/main amd64 python3-decorator all 4.0.6-1 [12.8 kB]                                                 
 Get:9 http://ftp.us.debian.org/debian testing/main amd64 python3-pytools all 2016.2.1-1 [33.9 kB]                                                
 Get:10 http://ftp.us.debian.org/debian testing/contrib amd64 python3-pycuda amd64 2016.1-1+b2 [298 kB]                                              
 Fetched 7,150 kB in 1min 23s (85.4 kB/s)                                                                             
 Selecting previously unselected package fonts-mathjax.  
 (Reading database ... 281119 files and directories currently installed.)  
 Preparing to unpack .../fonts-mathjax_2.6.1-1_all.deb ...  
 Unpacking fonts-mathjax (2.6.1-1) ...  
 Selecting previously unselected package libboost-python1.61.0.  
 Preparing to unpack .../libboost-python1.61.0_1.61.0+dfsg-2.1_amd64.deb ...  
 Unpacking libboost-python1.61.0 (1.61.0+dfsg-2.1) ...  
 Selecting previously unselected package libboost-system1.61.0:amd64.  
 Preparing to unpack .../libboost-system1.61.0_1.61.0+dfsg-2.1_amd64.deb ...  
 Unpacking libboost-system1.61.0:amd64 (1.61.0+dfsg-2.1) ...  
 Selecting previously unselected package libboost-thread1.61.0:amd64.  
 Preparing to unpack .../libboost-thread1.61.0_1.61.0+dfsg-2.1_amd64.deb ...  
 Unpacking libboost-thread1.61.0:amd64 (1.61.0+dfsg-2.1) ...  
 Selecting previously unselected package libjs-mathjax.  
 Preparing to unpack .../libjs-mathjax_2.6.1-1_all.deb ...  
 Unpacking libjs-mathjax (2.6.1-1) ...  
 Selecting previously unselected package python-pycuda-doc.  
 Preparing to unpack .../python-pycuda-doc_2016.1-1_all.deb ...  
 Unpacking python-pycuda-doc (2016.1-1) ...  
 Selecting previously unselected package python3-appdirs.  
 Preparing to unpack .../python3-appdirs_1.4.0-2_all.deb ...  
 Unpacking python3-appdirs (1.4.0-2) ...  
 Selecting previously unselected package python3-decorator.  
 Preparing to unpack .../python3-decorator_4.0.6-1_all.deb ...  
 Unpacking python3-decorator (4.0.6-1) ...  
 Selecting previously unselected package python3-pytools.  
 Preparing to unpack .../python3-pytools_2016.2.1-1_all.deb ...  
 Unpacking python3-pytools (2016.2.1-1) ...  
 Selecting previously unselected package python3-pycuda.  
 Preparing to unpack .../python3-pycuda_2016.1-1+b2_amd64.deb ...  
 Unpacking python3-pycuda (2016.1-1+b2) ...  
 Processing triggers for fontconfig (2.11.0-6.5) ...  
 Processing triggers for libc-bin (2.19-22) ...  
 Setting up fonts-mathjax (2.6.1-1) ...  
 Setting up libboost-python1.61.0 (1.61.0+dfsg-2.1) ...  
 Setting up libboost-system1.61.0:amd64 (1.61.0+dfsg-2.1) ...  
 Setting up libboost-thread1.61.0:amd64 (1.61.0+dfsg-2.1) ...  
 Setting up libjs-mathjax (2.6.1-1) ...  
 Setting up python-pycuda-doc (2016.1-1) ...  
 Setting up python3-appdirs (1.4.0-2) ...  
 Setting up python3-decorator (4.0.6-1) ...  
 Setting up python3-pytools (2016.2.1-1) ...  
 Setting up python3-pycuda (2016.1-1+b2) ...  
 Processing triggers for libc-bin (2.19-22) ...  

Okay, we are all good. Let's start python3 interpreter. By the way, I'm using python3.5

 user@localhost:~$ python3  
 Python 3.5.2+ (default, Aug 5 2016, 08:07:14)   
 [GCC 6.1.1 20160724] on linux  
 Type "help", "copyright", "credits" or "license" for more information.  
 >>> import pycuda.autoinit  
 Traceback (most recent call last):  
  File "<stdin>", line 1, in <module>  
  File "/usr/lib/python3/dist-packages/pycuda/autoinit.py", line 5, in <module>  
   cuda.init()  
 pycuda._driver.RuntimeError: cuInit failed: no CUDA-capable device is detected  
   

ah craps, you would think that something is wrong with the lib. It's just that the library did not detect a gpu that is cuda capable. For your information, I have workstation that has two gpu, an intel and nvidia gpu, so it is currently running intel which is not power consumption intensive and I have to explicitly enable nvidia gpu should I need to. With that said, let's try it again.

 user@localhost:~$ optirun python3  
 Python 3.5.2+ (default, Aug 5 2016, 08:07:14)   
 [GCC 6.1.1 20160724] on linux  
 Type "help", "copyright", "credits" or "license" for more information.  
 >>> import pycuda.autoinit  
 >>> import pycuda.driver as drv  
 >>> import numpy  
 >>>   
 >>> from pycuda.compiler import SourceModule  
 >>> mod = SourceModule("""  
 ... __global__ void multiply_them(float *dest, float *a, float *b)  
 ... {  
 ...   const int i = threadIdx.x;  
 ...   dest[i] = a[i] * b[i];  
 ... }  
 ... """)  
 >>> multiple_them = mod.get_function("multiply_them")  
 >>> a = numpy.random.randn(400).astype(numpy.float32)  
 >>> b = numpy.random.randn(400).astype(numpy.float32)  
 >>>   
 >>> dest = numpy.zeros_like(a)  
 >>> multiple_them(drv.Out(dest), drv.In(a), drv.In(b), block=(400,1,1), grid=(1,1))  
 >>> print(dest-a*b)  
 [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.  
  0. 0. 0. 0.]  

optirun is a command to enable nvidia discreet gpu on debian. So now import library and it works! Brilliant. By the way, I'm using nvidia 960M gpu. That's it for today.

Friday, June 2, 2017

First learning into gpu sort with code compile on gpuqsortlib

Many articles recently showing using gpu to do arithmetic operation and with so many chips on a gpu in comparison with cpu, I thought it is worth while to spend sometime to look into gpu capabilities. Today I'm trying out gpu sort with gpusortlib.

Download the gpusortlib and extract it. Then let's start make command

 user@localhost:~/Desktop/gpuqsortlib$ make   
 nvcc -O3 -I/common/inc -Xcompiler -fPIC -Iinc -c src/gpuqsort.cu -o gpuqsort.o  
 make: nvcc: Command not found  
 Makefile:19: recipe for target 'gpuqsort.o' failed  
 make: *** [gpuqsort.o] Error 127  

nvcc compiler not install, so let's install it. Note that, this required 668MB of disk space!

 user@localhost:~/Desktop/gpuqsortlib$ sudo apt-get install nvidia-cuda-toolkit  
 Reading package lists... Done  
 Building dependency tree      
 Reading state information... Done  
 The following packages were automatically installed and are no longer required:  
  libgl1-nvidia-glx:i386 libgl1-nvidia-glx-i386:i386 libllvm3.5v5 libnvidia-glcore:i386 linux-image-4.1.0-2-amd64 linux-image-4.2.0-1-amd64 linux-source-4.3 python3-ecdsa syslinux  
  unetbootin-translations  
 Use 'sudo apt autoremove' to remove them.  
 The following additional packages will be installed:  
  libcublas7.5 libcuda1 libcuda1:i386 libcuda1-i386:i386 libcudart7.5 libcufft7.5 libcufftw7.5 libcuinj64-7.5 libcurand7.5 libcusolver7.5 libcusparse7.5 libegl1-nvidia libgl1-nvidia-glx  
  libgl1-nvidia-glx:i386 libgles1-nvidia libgles2-nvidia libnppc7.5 libnppi7.5 libnpps7.5 libnvblas7.5 libnvcuvid1 libnvidia-compiler libnvidia-eglcore libnvidia-glcore  
  libnvidia-glcore:i386 libnvidia-ml1 libnvidia-ml1:i386 libnvrtc7.5 libnvtoolsext1 libnvvm3 libthrust-dev libva-x11-1:i386 libvdpau-dev libvdpau-va-gl1:i386 libvdpau1 libvdpau1:i386  
  mesa-vdpau-drivers:i386 nvidia-alternative nvidia-cuda-dev nvidia-cuda-doc nvidia-cuda-gdb nvidia-driver nvidia-driver-bin nvidia-driver-libs nvidia-kernel-support nvidia-opencl-common  
  nvidia-opencl-dev nvidia-opencl-icd nvidia-profiler nvidia-smi:i386 nvidia-vdpau-driver nvidia-visual-profiler opencl-headers vdpau-driver-all:i386 xserver-xorg-video-nvidia  
 Suggested packages:  
  nvidia-cuda-mps nvidia-cuda-mps:i386 libvdpau-doc libcupti-dev nvidia-vdpau-driver:i386 nvidia-legacy-340xx-vdpau-driver:i386  
 Recommended packages:  
  nvidia-smi nvidia-settings nvidia-driver-libs-i386 libnvidia-cfg1  
 The following NEW packages will be installed:  
  libcublas7.5 libcuda1 libcuda1:i386 libcuda1-i386:i386 libcudart7.5 libcufft7.5 libcufftw7.5 libcuinj64-7.5 libcurand7.5 libcusolver7.5 libcusparse7.5 libnppc7.5 libnppi7.5 libnpps7.5  
  libnvblas7.5 libnvcuvid1 libnvidia-compiler libnvidia-glcore libnvidia-glcore:i386 libnvidia-ml1:i386 libnvrtc7.5 libnvtoolsext1 libnvvm3 libthrust-dev libva-x11-1:i386 libvdpau-dev  
  libvdpau-va-gl1:i386 mesa-vdpau-drivers:i386 nvidia-cuda-dev nvidia-cuda-doc nvidia-cuda-gdb nvidia-cuda-toolkit nvidia-driver-libs nvidia-opencl-common nvidia-opencl-dev  
  nvidia-opencl-icd nvidia-profiler nvidia-smi:i386 nvidia-visual-profiler opencl-headers vdpau-driver-all:i386  
 The following packages will be upgraded:  
  libegl1-nvidia libgl1-nvidia-glx libgl1-nvidia-glx:i386 libgles1-nvidia libgles2-nvidia libnvidia-eglcore libnvidia-ml1 libvdpau1 libvdpau1:i386 nvidia-alternative nvidia-driver  
  nvidia-driver-bin nvidia-kernel-support nvidia-vdpau-driver xserver-xorg-video-nvidia  
 15 upgraded, 41 newly installed, 0 to remove and 510 not upgraded.  
 Need to get 668 MB of archives.  
 After this operation, 1,329 MB of additional disk space will be used.  
 Do you want to continue? [Y/n] Y  
 Get:1 http://ftp.us.debian.org/debian testing/non-free amd64 libgles2-nvidia amd64 352.79-10 [170 kB]  
 Get:2 http://ftp.us.debian.org/debian testing/non-free amd64 libgles1-nvidia amd64 352.79-10 [165 kB]  
 Get:3 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-driver amd64 352.79-10 [435 kB]  
 Get:4 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-driver-bin amd64 352.79-10 [230 kB]  
 Get:5 http://ftp.us.debian.org/debian testing/non-free amd64 xserver-xorg-video-nvidia amd64 352.79-10 [3,188 kB]  
 Get:6 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-vdpau-driver amd64 352.79-10 [620 kB]                                            
 Get:7 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-kernel-support amd64 352.79-10 [150 kB]                                           
 Get:8 http://ftp.us.debian.org/debian testing/non-free amd64 libegl1-nvidia amd64 352.79-10 [291 kB]                                               
 Get:9 http://ftp.us.debian.org/debian testing/non-free amd64 libgl1-nvidia-glx amd64 352.79-10 [527 kB]                                             
 Get:10 http://ftp.us.debian.org/debian testing/non-free i386 libgl1-nvidia-glx i386 352.79-10 [518 kB]                                              
 Get:11 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-alternative amd64 352.79-10 [151 kB]                                            
 Get:12 http://ftp.us.debian.org/debian testing/non-free amd64 libnvidia-ml1 amd64 352.79-10 [452 kB]                                               
 Get:13 http://ftp.us.debian.org/debian testing/non-free i386 libnvidia-glcore i386 352.79-10 [6,549 kB]                                             
 Get:14 http://ftp.us.debian.org/debian testing/non-free amd64 libnvidia-glcore amd64 352.79-10 [6,841 kB]                                            
 Get:15 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-driver-libs amd64 352.79-10 [150 kB]                                            
 Get:16 http://ftp.us.debian.org/debian testing/main i386 libvdpau1 i386 1.1.1-3 [39.7 kB]                                                    
 Get:17 http://ftp.us.debian.org/debian testing/main amd64 libvdpau1 amd64 1.1.1-3 [41.1 kB]                                                   
 Get:18 http://ftp.us.debian.org/debian testing/non-free amd64 libnvidia-eglcore amd64 352.79-10 [6,715 kB]                                            
 Get:19 http://ftp.us.debian.org/debian testing/non-free i386 libcuda1 i386 352.79-10 [4,154 kB]                                                 
 Get:20 http://ftp.us.debian.org/debian testing/non-free amd64 libcuda1 amd64 352.79-10 [4,072 kB]                                                
 Get:21 http://ftp.us.debian.org/debian testing/non-free i386 libcuda1-i386 i386 352.79-10 [149 kB]                                                
 Get:22 http://ftp.us.debian.org/debian testing/non-free amd64 libnvcuvid1 amd64 352.79-10 [624 kB]                                                
 Get:23 http://ftp.us.debian.org/debian testing/non-free amd64 libnvidia-compiler amd64 352.79-10 [11.7 MB]                                            
 Get:24 http://ftp.us.debian.org/debian testing/non-free i386 libnvidia-ml1 i386 352.79-10 [426 kB]                                                
 Get:25 http://ftp.us.debian.org/debian testing/main i386 libva-x11-1 i386 1.7.1-2 [19.9 kB]                                                   
 Get:26 http://ftp.us.debian.org/debian testing/main amd64 libvdpau-dev amd64 1.1.1-3 [53.0 kB]                                                  
 Get:27 http://ftp.us.debian.org/debian testing/main i386 libvdpau-va-gl1 i386 0.4.0-2 [73.0 kB]                                                 
 Get:28 http://ftp.us.debian.org/debian testing/main i386 mesa-vdpau-drivers i386 11.2.2-1 [1,669 kB]                                               
 Get:29 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-opencl-common amd64 352.79-10 [150 kB]                                           
 Get:30 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-opencl-icd amd64 352.79-10 [4,141 kB]                                            
 Get:31 http://ftp.us.debian.org/debian testing/non-free i386 nvidia-smi i386 352.79-10 [301 kB]                                                 
 Get:32 http://ftp.us.debian.org/debian testing/main i386 vdpau-driver-all i386 1.1.1-3 [20.2 kB]                                                 
 Get:33 http://ftp.us.debian.org/debian testing/non-free amd64 libcublas7.5 amd64 7.5.18-3 [14.1 MB]                                               
 Get:34 http://ftp.us.debian.org/debian testing/non-free amd64 libcudart7.5 amd64 7.5.18-3 [113 kB]                                                
 Get:35 http://ftp.us.debian.org/debian testing/non-free amd64 libcufft7.5 amd64 7.5.18-3 [66.6 MB]                                                
 Get:36 http://ftp.us.debian.org/debian testing/non-free amd64 libcufftw7.5 amd64 7.5.18-3 [126 kB]                                                
 Get:37 http://ftp.us.debian.org/debian testing/non-free amd64 libcuinj64-7.5 amd64 7.5.18-3 [1,651 kB]                                              
 Get:38 http://ftp.us.debian.org/debian testing/non-free amd64 libcurand7.5 amd64 7.5.18-3 [37.8 MB]                                               
 Get:39 http://ftp.us.debian.org/debian testing/non-free amd64 libcusolver7.5 amd64 7.5.18-3 [16.0 MB]                                              
 Get:40 http://ftp.us.debian.org/debian testing/non-free amd64 libcusparse7.5 amd64 7.5.18-3 [19.2 MB]                                              
 Get:41 http://ftp.us.debian.org/debian testing/non-free amd64 libnppc7.5 amd64 7.5.18-3 [118 kB]                                                 
 Get:42 http://ftp.us.debian.org/debian testing/non-free amd64 libnppi7.5 amd64 7.5.18-3 [30.6 MB]                                                
 Get:43 http://ftp.us.debian.org/debian testing/non-free amd64 libnpps7.5 amd64 7.5.18-3 [2,667 kB]                                                
 Get:44 http://ftp.us.debian.org/debian testing/non-free amd64 libnvblas7.5 amd64 7.5.18-3 [124 kB]                                                
 Get:45 http://ftp.us.debian.org/debian testing/non-free amd64 libnvrtc7.5 amd64 7.5.18-3 [6,254 kB]                                               
 Get:46 http://ftp.us.debian.org/debian testing/non-free amd64 libnvtoolsext1 amd64 7.5.18-3 [41.3 kB]                                              
 Get:47 http://ftp.us.debian.org/debian testing/non-free amd64 libnvvm3 amd64 7.5.18-3 [4,203 kB]                                                 
 Get:48 http://ftp.us.debian.org/debian testing/main amd64 libthrust-dev all 1.8.1-1 [439 kB]                                                   
 Get:49 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-cuda-dev amd64 7.5.18-3 [201 MB]                                              
 Get:50 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-cuda-doc all 7.5.18-3 [73.5 MB]                                               
 Get:51 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-cuda-gdb amd64 7.5.18-3 [2,613 kB]                                             
 Get:52 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-profiler amd64 7.5.18-3 [2,253 kB]                                             
 Get:53 http://ftp.us.debian.org/debian testing/main amd64 opencl-headers all 2.0~svn32091-2 [57.1 kB]                                              
 Get:54 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-opencl-dev amd64 7.5.18-3 [28.2 kB]                                             
 Get:55 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-cuda-toolkit amd64 7.5.18-3 [15.3 MB]                                            
 Get:56 http://ftp.us.debian.org/debian testing/non-free amd64 nvidia-visual-profiler amd64 7.5.18-3 [118 MB]                                           
 Fetched 668 MB in 33min 21s (334 kB/s)                                                                              
 Reading changelogs... Done  
 Extracting templates from packages: 100%  
 (Reading database ... 275391 files and directories currently installed.)  
 Preparing to unpack .../libgles2-nvidia_352.79-10_amd64.deb ...  
 Unpacking libgles2-nvidia:amd64 (352.79-10) over (352.79-8) ...  
 Preparing to unpack .../libgles1-nvidia_352.79-10_amd64.deb ...  
 Unpacking libgles1-nvidia:amd64 (352.79-10) over (352.79-8) ...  
 Preparing to unpack .../nvidia-driver_352.79-10_amd64.deb ...  
 Unpacking nvidia-driver (352.79-10) over (352.79-8) ...  
 Preparing to unpack .../nvidia-driver-bin_352.79-10_amd64.deb ...  
 Unpacking nvidia-driver-bin (352.79-10) over (352.79-8) ...  
 Preparing to unpack .../xserver-xorg-video-nvidia_352.79-10_amd64.deb ...  
 Unpacking xserver-xorg-video-nvidia (352.79-10) over (352.79-8) ...  
 Preparing to unpack .../nvidia-vdpau-driver_352.79-10_amd64.deb ...  
 Unpacking nvidia-vdpau-driver:amd64 (352.79-10) over (352.79-8) ...  
 Preparing to unpack .../nvidia-kernel-support_352.79-10_amd64.deb ...  
 Unpacking nvidia-kernel-support (352.79-10) over (352.79-8) ...  
 Preparing to unpack .../libegl1-nvidia_352.79-10_amd64.deb ...  
 Unpacking libegl1-nvidia:amd64 (352.79-10) over (352.79-8) ...  
 Preparing to unpack .../libgl1-nvidia-glx_352.79-10_i386.deb ...  
 De-configuring libgl1-nvidia-glx:amd64 (352.79-8) ...  
 Unpacking libgl1-nvidia-glx:i386 (352.79-10) over (352.79-8) ...  
 Preparing to unpack .../libgl1-nvidia-glx_352.79-10_amd64.deb ...  
 Unpacking libgl1-nvidia-glx:amd64 (352.79-10) over (352.79-8) ...  
 Preparing to unpack .../nvidia-alternative_352.79-10_amd64.deb ...  
 Unpacking nvidia-alternative (352.79-10) over (352.79-8) ...  
 Preparing to unpack .../libnvidia-ml1_352.79-10_amd64.deb ...  
 Unpacking libnvidia-ml1:amd64 (352.79-10) over (352.79-8) ...  
 Selecting previously unselected package libnvidia-glcore:i386.  
 Preparing to unpack .../libnvidia-glcore_352.79-10_i386.deb ...  
 Unpacking libnvidia-glcore:i386 (352.79-10) ...  
 Selecting previously unselected package libnvidia-glcore:amd64.  
 Preparing to unpack .../libnvidia-glcore_352.79-10_amd64.deb ...  
 Unpacking libnvidia-glcore:amd64 (352.79-10) ...  
 Selecting previously unselected package nvidia-driver-libs:amd64.  
 Preparing to unpack .../nvidia-driver-libs_352.79-10_amd64.deb ...  
 Unpacking nvidia-driver-libs:amd64 (352.79-10) ...  
 Preparing to unpack .../libvdpau1_1.1.1-3_amd64.deb ...  
 De-configuring libvdpau1:i386 (1.1.1-1) ...  
 Unpacking libvdpau1:amd64 (1.1.1-3) over (1.1.1-1) ...  
 Preparing to unpack .../libvdpau1_1.1.1-3_i386.deb ...  
 Unpacking libvdpau1:i386 (1.1.1-3) over (1.1.1-1) ...  
 Preparing to unpack .../libnvidia-eglcore_352.79-10_amd64.deb ...  
 Unpacking libnvidia-eglcore:amd64 (352.79-10) over (352.79-8) ...  
 Selecting previously unselected package libcuda1:amd64.  
 Preparing to unpack .../libcuda1_352.79-10_amd64.deb ...  
 Unpacking libcuda1:amd64 (352.79-10) ...  
 Selecting previously unselected package libcuda1:i386.  
 Preparing to unpack .../libcuda1_352.79-10_i386.deb ...  
 Unpacking libcuda1:i386 (352.79-10) ...  
 Selecting previously unselected package libcuda1-i386:i386.  
 Preparing to unpack .../libcuda1-i386_352.79-10_i386.deb ...  
 Unpacking libcuda1-i386:i386 (352.79-10) ...  
 Selecting previously unselected package libnvcuvid1:amd64.  
 Preparing to unpack .../libnvcuvid1_352.79-10_amd64.deb ...  
 Unpacking libnvcuvid1:amd64 (352.79-10) ...  
 Selecting previously unselected package libnvidia-compiler:amd64.  
 Preparing to unpack .../libnvidia-compiler_352.79-10_amd64.deb ...  
 Unpacking libnvidia-compiler:amd64 (352.79-10) ...  
 Selecting previously unselected package libnvidia-ml1:i386.  
 Preparing to unpack .../libnvidia-ml1_352.79-10_i386.deb ...  
 Unpacking libnvidia-ml1:i386 (352.79-10) ...  
 Selecting previously unselected package libva-x11-1:i386.  
 Preparing to unpack .../libva-x11-1_1.7.1-2_i386.deb ...  
 Unpacking libva-x11-1:i386 (1.7.1-2) ...  
 Selecting previously unselected package libvdpau-dev:amd64.  
 Preparing to unpack .../libvdpau-dev_1.1.1-3_amd64.deb ...  
 Unpacking libvdpau-dev:amd64 (1.1.1-3) ...  
 Selecting previously unselected package libvdpau-va-gl1:i386.  
 Preparing to unpack .../libvdpau-va-gl1_0.4.0-2_i386.deb ...  
 Unpacking libvdpau-va-gl1:i386 (0.4.0-2) ...  
 Selecting previously unselected package mesa-vdpau-drivers:i386.  
 Preparing to unpack .../mesa-vdpau-drivers_11.2.2-1_i386.deb ...  
 Unpacking mesa-vdpau-drivers:i386 (11.2.2-1) ...  
 Selecting previously unselected package nvidia-opencl-common.  
 Preparing to unpack .../nvidia-opencl-common_352.79-10_amd64.deb ...  
 Unpacking nvidia-opencl-common (352.79-10) ...  
 Selecting previously unselected package nvidia-opencl-icd:amd64.  
 Preparing to unpack .../nvidia-opencl-icd_352.79-10_amd64.deb ...  
 Unpacking nvidia-opencl-icd:amd64 (352.79-10) ...  
 Selecting previously unselected package nvidia-smi:i386.  
 Preparing to unpack .../nvidia-smi_352.79-10_i386.deb ...  
 Unpacking nvidia-smi:i386 (352.79-10) ...  
 Selecting previously unselected package vdpau-driver-all:i386.  
 Preparing to unpack .../vdpau-driver-all_1.1.1-3_i386.deb ...  
 Unpacking vdpau-driver-all:i386 (1.1.1-3) ...  
 Selecting previously unselected package libcublas7.5:amd64.  
 Preparing to unpack .../libcublas7.5_7.5.18-3_amd64.deb ...  
 Unpacking libcublas7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libcudart7.5:amd64.  
 Preparing to unpack .../libcudart7.5_7.5.18-3_amd64.deb ...  
 Unpacking libcudart7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libcufft7.5:amd64.  
 Preparing to unpack .../libcufft7.5_7.5.18-3_amd64.deb ...  
 Unpacking libcufft7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libcufftw7.5:amd64.  
 Preparing to unpack .../libcufftw7.5_7.5.18-3_amd64.deb ...  
 Unpacking libcufftw7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libcuinj64-7.5:amd64.  
 Preparing to unpack .../libcuinj64-7.5_7.5.18-3_amd64.deb ...  
 Unpacking libcuinj64-7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libcurand7.5:amd64.  
 Preparing to unpack .../libcurand7.5_7.5.18-3_amd64.deb ...  
 Unpacking libcurand7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libcusolver7.5:amd64.  
 Preparing to unpack .../libcusolver7.5_7.5.18-3_amd64.deb ...  
 Unpacking libcusolver7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libcusparse7.5:amd64.  
 Preparing to unpack .../libcusparse7.5_7.5.18-3_amd64.deb ...  
 Unpacking libcusparse7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libnppc7.5:amd64.  
 Preparing to unpack .../libnppc7.5_7.5.18-3_amd64.deb ...  
 Unpacking libnppc7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libnppi7.5:amd64.  
 Preparing to unpack .../libnppi7.5_7.5.18-3_amd64.deb ...  
 Unpacking libnppi7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libnpps7.5:amd64.  
 Preparing to unpack .../libnpps7.5_7.5.18-3_amd64.deb ...  
 Unpacking libnpps7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libnvblas7.5:amd64.  
 Preparing to unpack .../libnvblas7.5_7.5.18-3_amd64.deb ...  
 Unpacking libnvblas7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libnvrtc7.5:amd64.  
 Preparing to unpack .../libnvrtc7.5_7.5.18-3_amd64.deb ...  
 Unpacking libnvrtc7.5:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libnvtoolsext1:amd64.  
 Preparing to unpack .../libnvtoolsext1_7.5.18-3_amd64.deb ...  
 Unpacking libnvtoolsext1:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libnvvm3:amd64.  
 Preparing to unpack .../libnvvm3_7.5.18-3_amd64.deb ...  
 Unpacking libnvvm3:amd64 (7.5.18-3) ...  
 Selecting previously unselected package libthrust-dev.  
 Preparing to unpack .../libthrust-dev_1.8.1-1_all.deb ...  
 Unpacking libthrust-dev (1.8.1-1) ...  
 Selecting previously unselected package nvidia-cuda-dev.  
 Preparing to unpack .../nvidia-cuda-dev_7.5.18-3_amd64.deb ...  
 Unpacking nvidia-cuda-dev (7.5.18-3) ...  
 Selecting previously unselected package nvidia-cuda-doc.  
 Preparing to unpack .../nvidia-cuda-doc_7.5.18-3_all.deb ...  
 Unpacking nvidia-cuda-doc (7.5.18-3) ...  
 Selecting previously unselected package nvidia-cuda-gdb.  
 Preparing to unpack .../nvidia-cuda-gdb_7.5.18-3_amd64.deb ...  
 Unpacking nvidia-cuda-gdb (7.5.18-3) ...  
 Selecting previously unselected package nvidia-profiler.  
 Preparing to unpack .../nvidia-profiler_7.5.18-3_amd64.deb ...  
 Unpacking nvidia-profiler (7.5.18-3) ...  
 Selecting previously unselected package opencl-headers.  
 Preparing to unpack .../opencl-headers_2.0~svn32091-2_all.deb ...  
 Unpacking opencl-headers (2.0~svn32091-2) ...  
 Selecting previously unselected package nvidia-opencl-dev:amd64.  
 Preparing to unpack .../nvidia-opencl-dev_7.5.18-3_amd64.deb ...  
 Unpacking nvidia-opencl-dev:amd64 (7.5.18-3) ...  
 Selecting previously unselected package nvidia-cuda-toolkit.  
 Preparing to unpack .../nvidia-cuda-toolkit_7.5.18-3_amd64.deb ...  
 Unpacking nvidia-cuda-toolkit (7.5.18-3) ...  
 Selecting previously unselected package nvidia-visual-profiler.  
 Preparing to unpack .../nvidia-visual-profiler_7.5.18-3_amd64.deb ...  
 Unpacking nvidia-visual-profiler (7.5.18-3) ...  
 Processing triggers for libc-bin (2.19-22) ...  
 Processing triggers for glx-alternative-nvidia (0.7.3) ...  
 Processing triggers for desktop-file-utils (0.23-1) ...  
 Processing triggers for gnome-menus (3.13.3-8) ...  
 Processing triggers for mime-support (3.60) ...  
 Processing triggers for man-db (2.7.5-1) ...  
 Processing triggers for update-glx (0.7.3) ...  
 Processing triggers for glx-alternative-nvidia (0.7.3) ...  
 update-alternatives: using /usr/lib/nvidia to provide /usr/lib/glx (glx) in auto mode  
 update-alternatives: warning: skip creation of /usr/lib/x86_64-linux-gnu/libEGL.so.1 because associated file /usr/lib/x86_64-linux-gnu/nvidia/libEGL.so.1 (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /usr/lib/i386-linux-gnu/libGL.so.1 because associated file /usr/lib/i386-linux-gnu/nvidia/libGL.so.1 (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /usr/lib/x86_64-linux-gnu/libGL.so.1 because associated file /usr/lib/x86_64-linux-gnu/nvidia/libGL.so.1 (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /usr/lib/x86_64-linux-gnu/libGLESv1_CM.so.1 because associated file /usr/lib/x86_64-linux-gnu/nvidia/libGLESv1_CM.so.1 (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /usr/lib/x86_64-linux-gnu/libGLESv2.so.2 because associated file /usr/lib/x86_64-linux-gnu/nvidia/libGLESv2.so.2 (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /usr/lib/i386-linux-gnu/libnvidia-cfg.so.1 because associated file /usr/lib/i386-linux-gnu/nvidia/libnvidia-cfg.so.1 (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.1 because associated file /usr/lib/x86_64-linux-gnu/nvidia/libnvidia-cfg.so.1 (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /usr/lib/xorg/modules/linux/libglx.so because associated file /usr/lib/nvidia/libglx.so (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /etc/modprobe.d/nvidia-blacklists-nouveau.conf because associated file /etc/nvidia/nvidia-blacklists-nouveau.conf (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /usr/bin/nvidia-bug-report.sh because associated file /usr/lib/nvidia/nvidia-bug-report.sh (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /usr/share/X11/xorg.conf.d/nvidia-drm-outputclass.conf because associated file /etc/nvidia/nvidia-drm-outputclass.conf (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /etc/modules-load.d/nvidia.conf because associated file /etc/nvidia/nvidia-load.conf (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /etc/modprobe.d/nvidia.conf because associated file /etc/nvidia/nvidia-modprobe.conf (of link group glx) doesn't exist  
 update-alternatives: warning: skip creation of /usr/lib/xorg/modules/drivers/nvidia_drv.so because associated file /usr/lib/nvidia/nvidia_drv.so (of link group glx) doesn't exist  
 update-alternatives: using /usr/lib/mesa-diverted to provide /usr/lib/glx (glx) in auto mode  
 Processing triggers for libc-bin (2.19-22) ...  
 Processing triggers for initramfs-tools (0.120) ...  
 update-initramfs: Generating /boot/initrd.img-4.6.0-1-amd64  
 Processing triggers for bumblebee (3.2.1-13) ...  
 Setting up nvidia-alternative (352.79-10) ...  
 Setting up libnvidia-eglcore:amd64 (352.79-10) ...  
 Setting up libnvidia-glcore:amd64 (352.79-10) ...  
 Setting up libnvidia-glcore:i386 (352.79-10) ...  
 Setting up libvdpau1:amd64 (1.1.1-3) ...  
 Setting up libvdpau1:i386 (1.1.1-3) ...  
 Setting up libnvidia-compiler:amd64 (352.79-10) ...  
 Setting up libva-x11-1:i386 (1.7.1-2) ...  
 Setting up libvdpau-dev:amd64 (1.1.1-3) ...  
 Setting up libvdpau-va-gl1:i386 (0.4.0-2) ...  
 Setting up mesa-vdpau-drivers:i386 (11.2.2-1) ...  
 Setting up nvidia-opencl-common (352.79-10) ...  
 Setting up vdpau-driver-all:i386 (1.1.1-3) ...  
 Setting up libcublas7.5:amd64 (7.5.18-3) ...  
 Setting up libcudart7.5:amd64 (7.5.18-3) ...  
 Setting up libcufft7.5:amd64 (7.5.18-3) ...  
 Setting up libcufftw7.5:amd64 (7.5.18-3) ...  
 Setting up libcurand7.5:amd64 (7.5.18-3) ...  
 Setting up libcusolver7.5:amd64 (7.5.18-3) ...  
 Setting up libcusparse7.5:amd64 (7.5.18-3) ...  
 Setting up libnppc7.5:amd64 (7.5.18-3) ...  
 Setting up libnppi7.5:amd64 (7.5.18-3) ...  
 Setting up libnpps7.5:amd64 (7.5.18-3) ...  
 Setting up libnvblas7.5:amd64 (7.5.18-3) ...  
 Setting up libnvrtc7.5:amd64 (7.5.18-3) ...  
 Setting up libnvtoolsext1:amd64 (7.5.18-3) ...  
 Setting up libnvvm3:amd64 (7.5.18-3) ...  
 Setting up libthrust-dev (1.8.1-1) ...  
 Setting up nvidia-cuda-doc (7.5.18-3) ...  
 Setting up nvidia-cuda-gdb (7.5.18-3) ...  
 Setting up opencl-headers (2.0~svn32091-2) ...  
 Setting up nvidia-opencl-dev:amd64 (7.5.18-3) ...  
 Processing triggers for nvidia-alternative (352.79-10) ...  
 update-alternatives: using /usr/lib/nvidia/current to provide /usr/lib/nvidia/nvidia (nvidia) in auto mode  
 Setting up libgles2-nvidia:amd64 (352.79-10) ...  
 Setting up libgles1-nvidia:amd64 (352.79-10) ...  
 Setting up libgl1-nvidia-glx:amd64 (352.79-10) ...  
 Setting up libgl1-nvidia-glx:i386 (352.79-10) ...  
 Setting up libegl1-nvidia:amd64 (352.79-10) ...  
 Setting up nvidia-driver-libs:amd64 (352.79-10) ...  
 Setting up libnvidia-ml1:amd64 (352.79-10) ...  
 Setting up libnvidia-ml1:i386 (352.79-10) ...  
 Setting up nvidia-driver-bin (352.79-10) ...  
 Setting up xserver-xorg-video-nvidia (352.79-10) ...  
 Setting up nvidia-vdpau-driver:amd64 (352.79-10) ...  
 Setting up nvidia-driver (352.79-10) ...  
 Setting up nvidia-kernel-support (352.79-10) ...  
 Setting up libcuda1:amd64 (352.79-10) ...  
 Setting up libcuda1:i386 (352.79-10) ...  
 Setting up libcuda1-i386:i386 (352.79-10) ...  
 Setting up libnvcuvid1:amd64 (352.79-10) ...  
 Setting up nvidia-opencl-icd:amd64 (352.79-10) ...  
 Setting up nvidia-smi:i386 (352.79-10) ...  
 Setting up libcuinj64-7.5:amd64 (7.5.18-3) ...  
 Setting up nvidia-cuda-dev (7.5.18-3) ...  
 Setting up nvidia-profiler (7.5.18-3) ...  
 Setting up nvidia-cuda-toolkit (7.5.18-3) ...  
 Setting up nvidia-visual-profiler (7.5.18-3) ...  
 Processing triggers for libc-bin (2.19-22) ...  
 Processing triggers for glx-alternative-nvidia (0.7.3) ...  
 Processing triggers for update-glx (0.7.3) ...  
 Processing triggers for glx-alternative-nvidia (0.7.3) ...  
 update-alternatives: using /usr/lib/nvidia to provide /usr/lib/glx (glx) in auto mode  
 update-alternatives: using /usr/lib/nvidia/bumblebee to provide /usr/lib/glx (glx) in auto mode  
 Processing triggers for libc-bin (2.19-22) ...  
 Processing triggers for initramfs-tools (0.120) ...  
 update-initramfs: Generating /boot/initrd.img-4.6.0-1-amd64  
 Processing triggers for bumblebee (3.2.1-13) ...  

Let's run the make command again.

 user@localhost:~/Desktop/gpuqsortlib$ make  
 nvcc -O3 -I/common/inc -Xcompiler -fPIC -Iinc -c src/gpuqsort.cu -o gpuqsort.o  
 src/gpuqsort.cu(429): warning: conversion from a string literal to "char *" is deprecated  
   
 src/gpuqsort.cu(429): warning: conversion from a string literal to "char *" is deprecated  
   
 mkdir -p lib/linux/static  
 mkdir -p lib/linux/shared  
 nvcc -o lib/linux/static/libgpuqsort.a -lib gpuqsort.o  
 nvcc -o lib/linux/shared/libgpuqsort.so -shared gpuqsort.o  
 gcc -O3 -Iinc -c testbench/main.cpp -o main.o  
 testbench/main.cpp:33:21: fatal error: sqlite3.h: No such file or directory  
  #include "sqlite3.h"  
            ^  
 compilation terminated.  
 Makefile:22: recipe for target 'main.o' failed  
 make: *** [main.o] Error 1  

So sqlite3 header is not install, you can comment out define HASSQLITE3 in testbench/defs.h or install it. For me, I have install it

 user@localhost:~/Desktop/gpuqsortlib$ sudo apt-get install libsqlite3-dev  
 Reading package lists... Done  
 Building dependency tree      
 Reading state information... Done  
 The following packages were automatically installed and are no longer required:  
  libgl1-nvidia-glx:i386 libgl1-nvidia-glx-i386:i386 libllvm3.5v5 libnvidia-glcore:i386 linux-image-4.1.0-2-amd64 linux-image-4.2.0-1-amd64 linux-source-4.3 python3-ecdsa syslinux  
  unetbootin-translations  
 Use 'sudo apt autoremove' to remove them.  
 Suggested packages:  
  sqlite3-doc  
 The following NEW packages will be installed:  
  libsqlite3-dev  
 0 upgraded, 1 newly installed, 0 to remove and 510 not upgraded.  
 Need to get 689 kB of archives.  
 After this operation, 2,041 kB of additional disk space will be used.  
 Get:1 http://ftp.us.debian.org/debian testing/main amd64 libsqlite3-dev amd64 3.14.1-1 [689 kB]  
 Fetched 689 kB in 2s (256 kB/s)             
 Selecting previously unselected package libsqlite3-dev:amd64.  
 (Reading database ... 280685 files and directories currently installed.)  
 Preparing to unpack .../libsqlite3-dev_3.14.1-1_amd64.deb ...  
 Unpacking libsqlite3-dev:amd64 (3.14.1-1) ...  
 Setting up libsqlite3-dev:amd64 (3.14.1-1) ...  
   

and try again.

 user@localhost:~/Desktop/gpuqsortlib$ make  
 mkdir -p lib/linux/static  
 mkdir -p lib/linux/shared  
 nvcc -o lib/linux/static/libgpuqsort.a -lib gpuqsort.o  
 nvcc -o lib/linux/shared/libgpuqsort.so -shared gpuqsort.o  
 mkdir -p bin/linux  
 nvcc -o bin/linux/testbench main.o dists.o -lgpuqsort -lglut -L./lib/linux/static -lsqlite3  
 /usr/bin/ld: cannot find -lglut  
 collect2: error: ld returned 1 exit status  
 Makefile:15: recipe for target 'testbench' failed  
 make: *** [testbench] Error 1  
 user@localhost:~/Desktop/gpuqsortlib$ nvcc --version  
 nvcc: NVIDIA (R) Cuda compiler driver  
 Copyright (c) 2005-2015 NVIDIA Corporation  
 Built on Tue_Aug_11_14:27:32_CDT_2015  
 Cuda compilation tools, release 7.5, V7.5.17  

so the library glut (opengl utility toolkit) is not install,

 $ sudo apt-get install freeglut3-dev  
 Reading package lists... Done  
 Building dependency tree      
 Reading state information... Done  
 The following packages were automatically installed and are no longer required:  
  libgl1-nvidia-glx:i386 libgl1-nvidia-glx-i386:i386 libllvm3.5v5 libnvidia-glcore:i386 linux-image-4.1.0-2-amd64 linux-image-4.2.0-1-amd64 linux-source-4.3 python3-ecdsa syslinux  
  unetbootin-translations  
 Use 'sudo apt autoremove' to remove them.  
 The following additional packages will be installed:  
  freeglut3 libice-dev libsm-dev libxt-dev  
 Suggested packages:  
  libice-doc libsm-doc libxt-doc  
 The following NEW packages will be installed:  
  freeglut3 freeglut3-dev libice-dev libsm-dev libxt-dev  
 0 upgraded, 5 newly installed, 0 to remove and 510 not upgraded.  
 Need to get 826 kB of archives.  
 After this operation, 2,941 kB of additional disk space will be used.  
 Do you want to continue? [Y/n] Y  
 Get:1 http://ftp.us.debian.org/debian testing/main amd64 freeglut3 amd64 2.8.1-2 [124 kB]  
 Get:2 http://ftp.us.debian.org/debian testing/main amd64 libice-dev amd64 2:1.0.9-1+b1 [66.3 kB]  
 Get:3 http://ftp.us.debian.org/debian testing/main amd64 libsm-dev amd64 2:1.2.2-1+b1 [35.8 kB]  
 Get:4 http://ftp.us.debian.org/debian testing/main amd64 libxt-dev amd64 1:1.1.5-1 [423 kB]  
 Get:5 http://ftp.us.debian.org/debian testing/main amd64 freeglut3-dev amd64 2.8.1-2 [177 kB]  
 Fetched 826 kB in 2s (373 kB/s)      
 Selecting previously unselected package freeglut3:amd64.  
 (Reading database ... 280696 files and directories currently installed.)  
 Preparing to unpack .../freeglut3_2.8.1-2_amd64.deb ...  
 Unpacking freeglut3:amd64 (2.8.1-2) ...  
 Selecting previously unselected package libice-dev:amd64.  
 Preparing to unpack .../libice-dev_2%3a1.0.9-1+b1_amd64.deb ...  
 Unpacking libice-dev:amd64 (2:1.0.9-1+b1) ...  
 Selecting previously unselected package libsm-dev:amd64.  
 Preparing to unpack .../libsm-dev_2%3a1.2.2-1+b1_amd64.deb ...  
 Unpacking libsm-dev:amd64 (2:1.2.2-1+b1) ...  
 Selecting previously unselected package libxt-dev:amd64.  
 Preparing to unpack .../libxt-dev_1%3a1.1.5-1_amd64.deb ...  
 Unpacking libxt-dev:amd64 (1:1.1.5-1) ...  
 Selecting previously unselected package freeglut3-dev:amd64.  
 Preparing to unpack .../freeglut3-dev_2.8.1-2_amd64.deb ...  
 Unpacking freeglut3-dev:amd64 (2.8.1-2) ...  
 Processing triggers for man-db (2.7.5-1) ...  
 Setting up freeglut3:amd64 (2.8.1-2) ...  
 Setting up libice-dev:amd64 (2:1.0.9-1+b1) ...  
 Setting up libsm-dev:amd64 (2:1.2.2-1+b1) ...  
 Setting up libxt-dev:amd64 (1:1.1.5-1) ...  
 Setting up freeglut3-dev:amd64 (2.8.1-2) ...  
 Processing triggers for libc-bin (2.19-22) ...  

and one more try!

 user@localhost:~/Desktop/gpuqsortlib$ make  
 mkdir -p lib/linux/static  
 mkdir -p lib/linux/shared  
 nvcc -o lib/linux/static/libgpuqsort.a -lib gpuqsort.o  
 nvcc -o lib/linux/shared/libgpuqsort.so -shared gpuqsort.o  
 mkdir -p bin/linux  
 nvcc -o bin/linux/testbench main.o dists.o -lgpuqsort -lglut -L./lib/linux/static -lsqlite3  
 jason@JasonLaptop:~/Desktop/gpuqsortlib$ ./bin/linux/testbench   
   
 Usage:  
    testbench -[m|d|p] uniqueid  
 jason@JasonLaptop:~/Desktop/gpuqsortlib$ ./bin/linux/testbench -m 1  
 Testing 'models'  
 Unable to open file: norm-dragon.dat  

so finally got it compile and run fine.... but looks like there is no input sample file available! If you got beyond this point, leave your comment below and I would like to run on my workstation gpu too!

I have upload source code and compile code here.

Thursday, June 1, 2017

investigate into apache cassandra 1.2.19 sstable corrupt

Last, I have investigated in apache cassandra 1.0.8 sstable corruption a fresh node after upgraded to apache cassandra 1.2. Both of the articles can be found here and here. Today, we will take another look at a running apache cassandra 1.2.19 encounter sstable corruption. Below is the stack trace found in cassandra system.log

 org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.io.compress.CorruptBlockException: (/var/lib/cassandra/data/<KEYSPACE>/<COLUMN_FAMILY>/<KEYSPACE>-<CF>-ic-112-Data.db): corruption detected, chunk at 19042661 of length 27265.  
     at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:89)  
     at org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:45)  
     at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:355)  
     at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444)  
     at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424)  
     at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:380)  
     at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:391)  
     at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:370)  
     at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:175)  
     at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:155)  
     at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:142)  
     at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)  
     at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)  
     at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122)  
     at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96)  
     at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)  
     at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)  
     at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145)  
     at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)  
     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)  
     at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)  
     at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)  
     at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)  
     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)  
     at java.util.concurrent.FutureTask.run(FutureTask.java:262)  
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
     at java.lang.Thread.run(Thread.java:745)  
 Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/var/lib/cassandra/data/<KEYSPACE>/<COLUMN_FAMILY>/<KEYSPACE>-<CF>-ic-112-Data.db): corruption detected, chunk at 19042661 of length 27265.  
     at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:128)  
     at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:85)  
     ... 27 more  

Okay, let's trace into the stacktrace and study what actually cause this and what is sstable corruption means.

     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)  
     at java.util.concurrent.FutureTask.run(FutureTask.java:262)  
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
     at java.lang.Thread.run(Thread.java:745)      

Simple, a thread is run by the executor.

     at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145)  
     at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)  
     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)  
     at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)  
     at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)  
     at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)  

We see that a background compaction task is started. Code below.

   // the actual sstables to compact are not determined until we run the BCT; that way, if new sstables  
   // are created between task submission and execution, we execute against the most up-to-date information  
   class BackgroundCompactionTask implements Runnable  
   {  
     private final ColumnFamilyStore cfs;  
   
     BackgroundCompactionTask(ColumnFamilyStore cfs)  
     {  
       this.cfs = cfs;  
     }  
   
     public void run()  
     {  
       compactionLock.readLock().lock();  
       try  
       {  
         logger.debug("Checking {}.{}", cfs.table.name, cfs.columnFamily); // log after we get the lock so we can see delays from that if any  
         if (!cfs.isValid())  
         {  
           logger.debug("Aborting compaction for dropped CF");  
           return;  
         }  
   
         AbstractCompactionStrategy strategy = cfs.getCompactionStrategy();  
         AbstractCompactionTask task = strategy.getNextBackgroundTask(getDefaultGcBefore(cfs));  
         if (task == null)  
         {  
           logger.debug("No tasks available");  
           return;  
         }  
         task.execute(metrics);  
       }  
       finally  
       {  
         compactingCF.remove(cfs);  
         compactionLock.readLock().unlock();  
       }  
       submitBackground(cfs);  
     }  
   }  

     at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)  
     at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122)  
     at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96)  
     at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)  
     at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)  

Here we see an iterator going over the sstables for compaction.

     at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:175)  
     at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:155)  
     at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:142)  
     at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)  

DecoratedKey key = sstable.decodeKey(ByteBufferUtil.readWithShortLength(dfile));
     
Here we see that what actually get iterated is the sstable and particular on the key.

     at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:89)  
     at org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:45)  
     at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:355)  
     at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444)  
     at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424)  
     at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:380)  
     at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:391)  
     at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:370)  

Here, there is code reference has change in a few files due to method override, nonetheless, the important part on method reBuffer.


   @Override  
   protected void reBuffer()  
   {  
     try  
     {  
       decompressChunk(metadata.chunkFor(current));  
     }  
     catch (CorruptBlockException e)  
     {  
       throw new CorruptSSTableException(e, getPath());  
     }  
     catch (IOException e)  
     {  
       throw new FSReadError(e, getPath());  
     }  
   }  
     
   private void decompressChunk(CompressionMetadata.Chunk chunk) throws IOException  
   {  
     if (channel.position() != chunk.offset)  
       channel.position(chunk.offset);  
   
     if (compressed.capacity() < chunk.length)  
       compressed = ByteBuffer.wrap(new byte[chunk.length]);  
     else  
       compressed.clear();  
     compressed.limit(chunk.length);  
   
     if (channel.read(compressed) != chunk.length)  
       throw new CorruptBlockException(getPath(), chunk);  
   
     // technically flip() is unnecessary since all the remaining work uses the raw array, but if that changes  
     // in the future this will save a lot of hair-pulling  
     compressed.flip();  
     try  
     {  
       validBufferBytes = metadata.compressor().uncompress(compressed.array(), 0, chunk.length, buffer, 0);  
     }  
     catch (IOException e)  
     {  
       throw new CorruptBlockException(getPath(), chunk);  
     }  
   
     if (metadata.parameters.getCrcCheckChance() > FBUtilities.threadLocalRandom().nextDouble())  
     {  
       checksum.update(buffer, 0, validBufferBytes);  
   
       if (checksum(chunk) != (int) checksum.getValue())  
         throw new CorruptBlockException(getPath(), chunk);  
   
       // reset checksum object back to the original (blank) state  
       checksum.reset();  
     }  
   
     // buffer offset is always aligned  
     bufferOffset = current & ~(buffer.length - 1);  
   }  

we read that if chunk checksum is not the same as the updated crc32 checksum, this is consider sstable corruption.

For this type of exception, I remember I did many things such as below.

1. try online nodetool scrub, does not work
2. try offline sstablescrub, does not work.
3. wipeout the node and rebuild again, does not work.

we had to change the hardware altogether. Then we don't see the problem anymore.