My First Raspberry Pi 3 Model B

Hi friends,
I got my first Raspberry Pi 3 just yesterday i.e. 06.12.17. I did very little work concerned to Arduino till now. If you are interested you can visit my YouTube Channel at https://www.youtube.com/channel/UC_lrq5p2VNUtmgF2aynKIMA. All the content is free.

I have been working with Arduino from last year. I bought my first Arduino UNO in June, 2016, I forgot the exact day. I did very little work just as using sensors and analyzing the sensor based real time data through Python.

I also bought few other modules related to ARM processor along with RF and WiFi things. Internet of Things (IoT) is growing in leaps and bounds. I like Massimo Banzi's TED talk (https://www.youtube.com/watch?v=UoBUXOOdLXY), and also a couple of interviews related to Herman Hauser who is one of the person's behind ARM processor (https://www.youtube.com/watch?v=w1HODsDGMzI). We are living in very exciting times. The technology is growing in leaps and bounds. I could put-up a small lab in my home and office with help of all these modules and etc.

I started studying Electronical Design and Automation (EDA) along with Circuit Design and Simulation (CDS). I am excited to do all this through ngspice and KiCad and some stuff like that.  I loved this subject to that extent where I literally could work with these tools and could make very little stuff available for public through my YouTube Channel.

I came across NodeMCU and ESP32 just recently. micropython community is very active at these type of modules. I need to do lot of work in this area. Basically I am data scientist, my work related to IoT is all about dealing with data analysis and dashboards. Perhaps, I need to do a lot when it comes ML and AI in this area. OpenCV and TensorFlow communities are really rocking in this area which when coupled with Python do miracles. I guess...

MapReduce through Pydoop

Hi,

Today I am going to share my experience in running MapReduce wordcount program through pydoop (python) installed in UBUNTU 16.04.

I have been struggling with Hadoop for about a year. Now I am okay in running HDFS, MapReduce. However, yet to get thorough of the same.

I tried pydoop with my naive-like attitude. First I was trying to install pydoop in Ubuntu terminal (my favourite OS), but in vain. I wasted almost all day at my office. I could not figure out as how to erect pydoop since the office system works behind proxy :(

I came home and started doing the same in my PC (HP laptop). I first did the following statement in the terminal but found helpless...

sudo pip install pydoop. 

I found a problem regarding "HADOOP_HOME". I somehow understood that the pydoop is trying to figure-out Hadoop installation directly but fails. I got certain solution through forums  (promise, I forgot where it is....). I changed the command as below.

sudo pip -E install pydoop 

That "E" did the trick. Now pydoop set right properly in my laptop as I found from the output message.

As I already have Hadoop properly installed in pseudo-distribution mode (single node deployment). I did the following steps in terminal.

1. I created a small text file in which I have some arbitrary text.  
2. Uploaded to HDFS (directory)
3. Created two folder in HDFS directory such as ../input and ../output in the same directory
4. Wrote pydoop script file for word counting
5. Executed the script. 

My text file is as below: 

I created the folders in the one of the current directories in the HDFS.

hdfs dfs -ls /dir2
hdfs dfs -mkdir /dir2/input /dir2/output
hdfs dfs -copyFromLocal text.txt /dir2/input

Now I prepared the script file (script.py) with the following code.

def mapper(_, text, writer):
    for word in text.split():
        writer.emit(word, "1")

def reducer(word, icounts, writer):
    writer.emit(word, sum(map(int, icounts)))

Went back to the terminal and executed the following statement.

pydoop script script.py /dir2/input/text.txt /dir2/output/output_

Checked the results in the output folder of the directory.

BINGO!!! I am now a big data expert! Hey! Hey! Hey.

solved maven problem in Windows

I struggled a lot to install maven in windows 7 in my acer laptop. I could not execute mvn -version in CMD. I used to get the below error in CMD.

"org.codehaus.plexus.classworlds.launcher.ConfigurationException: Unhandled configuration (4)"

There was problem putting environment variables in system properties (Win+Pause/Break).

Initially I created two system variables i.e. MAVEN_HOME and M2_HOME. That was a mistake I guess. Then I deleted MAVEN_HOME and kept only M2_HOME. I updated M2_HOME in path variable as %M2_HOME% now it worked.

I executed C:\mvn -version it worked. 

C:\>mvn -version
C:\
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T22:11:4
7+05:30)
Maven home: C:\maven
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: C:\Java\jdk1.7.0_79\jre
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "x86", family: "windows"


C:\>

Now I got to see if I can build Hadoop with the help of maven!

How to know whether a package installed in R or not?

To know whether a package installed or not. You just have to execute the following command in console.

grep("abind", row.names(installed.packages()))

This will return 1. Because if you check the package "abind", it is the very first package name in the list of packages in right bottom pane in RStudio. In same way you can also check other packages, say I would like to check if package "semPlot" is installed or not.

> grep("semPlot", row.names(installed.packages()))
integer(0)

Observe the output message it is "integer(0)" which means the package with name "semPlot" is not installed.

How to install packages in RStudio through proxy server

I had this problem of installing packages through proxy server. I found certain useful code online on as how to install packages through http proxy. Do as below.

1. In RStudio; go to Tools -> Global Option -> Packages. Then uncheck both "Use secure download method for HTTP" and "Use Internet Explorer library/proxy for HTTP"
2. Go to the console (left bottom side pane) write file.edit("~/.Renviron") then enter. You will find small file named .Renviron in right bottom pane under "Files". 
3. Click the file .Renviron, now you will get the file opened at left upper pane as script file. Then input the following lines: 

options(internet.info = 0)

http_proxy="http://####:########@192.168.23.32:3128/"

http_proxy_user=ask

     4. Close RStudio and open again now try installing packages. 

How to save R output as in Excel/Calc/Spreadsheet tables?

I found a way to save R output into spreadsheets like Excel/Calc etc. R creates output as in lists when we perform regression. For example

> summary(s.manova)
             Df   Pillai approx F num Df den Df Pr(>F)
sc$gender     1 0.055830  0.43363      3     22 0.7311
sc$age        1 0.157079  1.36657      3     22 0.2790
sc$emp_level  1 0.099559  0.81082      3     22 0.5015
sc$salary     1 0.048873  0.37682      3     22 0.7706
sc$family     1 0.032379  0.24539      3     22 0.8637
Residuals    24                                      
> typeof(summary(s.manova))
[1] "list"

So if you observe the output it is a list. It might not be possible to export this chunk of output strait away as in the form of table in case if we are interested to use this output in reports. Usually in report we might have to put the results in the form of nice looking tables. But R could not create tables with vertical and horizontal lines separating cell wise values.

Do as below, in case if you are interested in converting such lists into nice looking tables with rows and columns for Word or Writer.

> write.csv(as.matrix(unlist(summary(s.manova)$stats)), "......./manova.ods")

In the above code "......./manova.ods" refers to your path to save the output with a name "manova.ods". Now it is possible to open the output file with the help of Libre Office Calc and edit the table as you like.

This method has one limitation that is we got to execute this step each time when we save table. That is next time if we are interested to save other table you got to repeat the same step but you may lose the previous table.

I will try to find solution as how to preserve all the output as in distinct tables in same spreadsheet next time.

Install Rgraphviz in R in Ubuntu

Recently I got to use Structural Equation Modelling for some certain article. I got to install psych to do SEM through sem. This package depends on Rgraphviz but Rgraphviz is not available for R 3.3. I found a useful resource online that shows as how to install Rgraphviz in R latest version. The following is the code.

source("http://bioconductor.org/biocLite.R")
biocLite("Rgraphviz")

Enjoy with psych now.