My First Raspberry Pi 3 Model B

Hi friends,
I got my first Raspberry Pi 3 just yesterday i.e. 06.12.17. I did very little work concerned to Arduino till now. If you are interested you can visit my YouTube Channel at https://www.youtube.com/channel/UC_lrq5p2VNUtmgF2aynKIMA. All the content is free.

I have been working with Arduino from last year. I bought my first Arduino UNO in June, 2016, I forgot the exact day. I did very little work just as using sensors and analyzing the sensor based real time data through Python.

I also bought few other modules related to ARM processor along with RF and WiFi things. Internet of Things (IoT) is growing in leaps and bounds. I like Massimo Banzi's TED talk (https://www.youtube.com/watch?v=UoBUXOOdLXY), and also a couple of interviews related to Herman Hauser who is one of the person's behind ARM processor (https://www.youtube.com/watch?v=w1HODsDGMzI). We are living in very exciting times. The technology is growing in leaps and bounds. I could put-up a small lab in my home and office with help of all these modules and etc.

I started studying Electronical Design and Automation (EDA) along with Circuit Design and Simulation (CDS). I am excited to do all this through ngspice and KiCad and some stuff like that.  I loved this subject to that extent where I literally could work with these tools and could make very little stuff available for public through my YouTube Channel.

I came across NodeMCU and ESP32 just recently. micropython community is very active at these type of modules. I need to do lot of work in this area. Basically I am data scientist, my work related to IoT is all about dealing with data analysis and dashboards. Perhaps, I need to do a lot when it comes ML and AI in this area. OpenCV and TensorFlow communities are really rocking in this area which when coupled with Python do miracles. I guess...

MapReduce through Pydoop

Hi,

Today I am going to share my experience in running MapReduce wordcount program through pydoop (python) installed in UBUNTU 16.04.

I have been struggling with Hadoop for about a year. Now I am okay in running HDFS, MapReduce. However, yet to get thorough of the same.

I tried pydoop with my naive-like attitude. First I was trying to install pydoop in Ubuntu terminal (my favourite OS), but in vain. I wasted almost all day at my office. I could not figure out as how to erect pydoop since the office system works behind proxy :(

I came home and started doing the same in my PC (HP laptop). I first did the following statement in the terminal but found helpless...

sudo pip install pydoop. 

I found a problem regarding "HADOOP_HOME". I somehow understood that the pydoop is trying to figure-out Hadoop installation directly but fails. I got certain solution through forums  (promise, I forgot where it is....). I changed the command as below.

sudo pip -E install pydoop 

That "E" did the trick. Now pydoop set right properly in my laptop as I found from the output message.

As I already have Hadoop properly installed in pseudo-distribution mode (single node deployment). I did the following steps in terminal.

1. I created a small text file in which I have some arbitrary text.  
2. Uploaded to HDFS (directory)
3. Created two folder in HDFS directory such as ../input and ../output in the same directory
4. Wrote pydoop script file for word counting
5. Executed the script. 

My text file is as below: 

I created the folders in the one of the current directories in the HDFS.

hdfs dfs -ls /dir2
hdfs dfs -mkdir /dir2/input /dir2/output
hdfs dfs -copyFromLocal text.txt /dir2/input

Now I prepared the script file (script.py) with the following code.

def mapper(_, text, writer):
    for word in text.split():
        writer.emit(word, "1")

def reducer(word, icounts, writer):
    writer.emit(word, sum(map(int, icounts)))

Went back to the terminal and executed the following statement.

pydoop script script.py /dir2/input/text.txt /dir2/output/output_

Checked the results in the output folder of the directory.

BINGO!!! I am now a big data expert! Hey! Hey! Hey.

solved maven problem in Windows

I struggled a lot to install maven in windows 7 in my acer laptop. I could not execute mvn -version in CMD. I used to get the below error in CMD.

"org.codehaus.plexus.classworlds.launcher.ConfigurationException: Unhandled configuration (4)"

There was problem putting environment variables in system properties (Win+Pause/Break).

Initially I created two system variables i.e. MAVEN_HOME and M2_HOME. That was a mistake I guess. Then I deleted MAVEN_HOME and kept only M2_HOME. I updated M2_HOME in path variable as %M2_HOME% now it worked.

I executed C:\mvn -version it worked. 

C:\>mvn -version
C:\
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T22:11:4
7+05:30)
Maven home: C:\maven
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: C:\Java\jdk1.7.0_79\jre
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "x86", family: "windows"


C:\>

Now I got to see if I can build Hadoop with the help of maven!

How to know whether a package installed in R or not?

To know whether a package installed or not. You just have to execute the following command in console.

grep("abind", row.names(installed.packages()))

This will return 1. Because if you check the package "abind", it is the very first package name in the list of packages in right bottom pane in RStudio. In same way you can also check other packages, say I would like to check if package "semPlot" is installed or not.

> grep("semPlot", row.names(installed.packages()))
integer(0)

Observe the output message it is "integer(0)" which means the package with name "semPlot" is not installed.

How to install packages in RStudio through proxy server

I had this problem of installing packages through proxy server. I found certain useful code online on as how to install packages through http proxy. Do as below.

1. In RStudio; go to Tools -> Global Option -> Packages. Then uncheck both "Use secure download method for HTTP" and "Use Internet Explorer library/proxy for HTTP"
2. Go to the console (left bottom side pane) write file.edit("~/.Renviron") then enter. You will find small file named .Renviron in right bottom pane under "Files". 
3. Click the file .Renviron, now you will get the file opened at left upper pane as script file. Then input the following lines: 

options(internet.info = 0)

http_proxy="http://####:########@192.168.23.32:3128/"

http_proxy_user=ask

     4. Close RStudio and open again now try installing packages. 

How to save R output as in Excel/Calc/Spreadsheet tables?

I found a way to save R output into spreadsheets like Excel/Calc etc. R creates output as in lists when we perform regression. For example

> summary(s.manova)
             Df   Pillai approx F num Df den Df Pr(>F)
sc$gender     1 0.055830  0.43363      3     22 0.7311
sc$age        1 0.157079  1.36657      3     22 0.2790
sc$emp_level  1 0.099559  0.81082      3     22 0.5015
sc$salary     1 0.048873  0.37682      3     22 0.7706
sc$family     1 0.032379  0.24539      3     22 0.8637
Residuals    24                                      
> typeof(summary(s.manova))
[1] "list"

So if you observe the output it is a list. It might not be possible to export this chunk of output strait away as in the form of table in case if we are interested to use this output in reports. Usually in report we might have to put the results in the form of nice looking tables. But R could not create tables with vertical and horizontal lines separating cell wise values.

Do as below, in case if you are interested in converting such lists into nice looking tables with rows and columns for Word or Writer.

> write.csv(as.matrix(unlist(summary(s.manova)$stats)), "......./manova.ods")

In the above code "......./manova.ods" refers to your path to save the output with a name "manova.ods". Now it is possible to open the output file with the help of Libre Office Calc and edit the table as you like.

This method has one limitation that is we got to execute this step each time when we save table. That is next time if we are interested to save other table you got to repeat the same step but you may lose the previous table.

I will try to find solution as how to preserve all the output as in distinct tables in same spreadsheet next time.

Install Rgraphviz in R in Ubuntu

Recently I got to use Structural Equation Modelling for some certain article. I got to install psych to do SEM through sem. This package depends on Rgraphviz but Rgraphviz is not available for R 3.3. I found a useful resource online that shows as how to install Rgraphviz in R latest version. The following is the code.

source("http://bioconductor.org/biocLite.R")
biocLite("Rgraphviz")

Enjoy with psych now.

Install R in Ubuntu 16.04

I installed Ubuntu 16.04 last month. I encountered errors while installing R in office desktop, I tried through CRAN mirror for India. But at home I tried 0-cloud and installed R without errors. I guess we need choose area specific cran-mirror.

Factor Rotation in R

There is always a question as what mapping technique that must be used while rotating factors. In fact, rotation is one of the mapping techniques along with other techniques like reflection, projection, scaling, shearing etc.

But it is always common to use rotation as preferred technique compared to other, the reason might be that it is simple to achieve. Rotation again two types, orthogonal and oblique. Which one we need to use for better factor solution? Gorsuch (1983, pp. 203-204) suggests orthogonal rotation if the factors under study are uncorrelated and oblique if the factors are correlated.

There are other studies that suggest the idea of simple structure despite of factor correlations. For instance, Thurstone (1947) proposes 5 rules to achieve simple structure. The rules are as follows (Brown, J. D., 2009): 

1. Each variable should produce at least one zero loading on some factor.
2. Each factor should have at least as many zero loadings as there are factors.
3. Each pair of factors should have variables with significant loadings on one and zero loadings on the other.
4. Each pair of factors should have a large proportion of zero loadings on both factors (if there are say four or more factors total).
5. Each pair of factors should have only a few complex variables. 

So, these two ideas i.e. from Gorsuch and Thurstone, might provide a clue regarding rotation technique. Now let us see how to handle with problem in R.

 

Install R very latest or current version in Ubuntu Linux

We may not be able to install the latest version R, when we install R through Software centre in Ubuntu. The following code is useful to install R latest version through terminal in Ubuntu.

1. First try to save the codename of Ubuntu by doing below in terminal.

codename=$(lsb_release -c -s)

Alternatively you can also try

lsb_release -c -s

to know the release name of Ubuntu in Terminal. What we are doing is that we are trying to save the version name in an object "codename", which we will use to update software sources.

2. Now update the software sources list by doing

echo "deb http://ftp.iitm.ac.in/cran/bin/linux/ubuntu $codename/" | sudo tee -a /etc/apt/sources.list > /dev/null  

Now you can open /etc/apt/sources.list through any text editor you may be able to find new line "deb http://ftp.iitm.ac.in/cran/bin/linux/ubuntu $codename/". You can use any other mirror of your interest. For CRAN mirrors please visit https://cran.r-project.org/mirrors.html.

3. Retrieve the apt key and add the apt-repository of R to the key-server.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9  
sudo add-apt-repository ppa:marutter/rdev

4. Update R by doing the below stepts.

sudo apt-get update  
sudo apt-get upgrade  
sudo apt-get install r-base r-base-dev

That's all you should be able to get new R version.

How to plot points on Indian map by using R Language

I started creating Maps through R. There are number of packages to take care of this. In this post I shall explain as how to fetch map with the help of  "ggmap" and then plot it with "ggplot2". 

We need a data file consisting coordinates (longitudes, latitudes) of any given terrain. In this post I am going to explain plotting points on Indian map. So, the steps are going to be: 

1. Preparing a data file with logitudes and latitudes
2. Fetching map from google server
3. Plotting points on the server

Step - 1: Preparing a data file with logitudes and latitudes: 

I already prepared a file which consists of coordinates and also a data column of accidents in different cities in India. You can download the file from here.  The data file is basically a CSV (comma separated values) file. It is easy to import CSV file in R (you may also import Excel files but it is not strait).  The following is the procedure to import the (CSV) file.

> my.df <- file.choose="" font="" read.csv="">
> my.df

my,df is my data set (which is basically an object) in which there is data column (variable) "accidents".  You may look at the variables (columns) by executing the following command in R editor. 

> edit(my.df)

Step - 2:  Fetching map from google server

The following procedure illustrate as how to get Indian map from google server. We need two packages I am going to invoke the libraries by using command "library"

> library(ggmap)
> library(ggplot2)
> my.map <- get_map="" india="" zoom="5)</span">
> ggmap(my.map)

You can see Indian map popping up in graph canvas adjacent to R console. The following is the graph retrieved from google server.        

Step - 3: Plotting points on the server

Now we are going to identify those cities (listed in our data file) where the accidents are identified. 

> ggmap(my.map)+geom_point(data=my.df, aes(x=my.df$Longitude, y=my.df$Latitude, label=my.df$acci))

We get the following map. 

But the points are not distinct, so let me make them little better by changing the color (red) and size (3.5).

> ggmap(my.map)+geom_point(data=my.df, aes(x=my.df$Longitude, y=my.df$Latitude, label=my.df$acci), color="red", size=3.5, alpha = 0.5)

The following is the map that I get with the help of above statement.

Now I guess it is better to get number of accidents just adjacent to the point. Isn't it? I can get the numbers just adjacent to the point by following statement.

> ggmap(my.map)+geom_point(data=my.df, aes(x=my.df$Longitude, y=my.df$Latitude, label=my.df$acci), color="red")+geom_text(data = my.df, aes(x = my.df$Longitude, y = my.df$Latitude, label = my.df$acci), size=3.5, vjust = 0.5, hjust = -0.75)

The following is the map that we can produce with above statement.

I hope you got the point!

How is it? Now, I am going to give you assignment! Yes, exactly.....
Delete the numbers and try to draw draw a line between any two given points on this map. You can also try on satellite map rather than a terrain map.

All the best!

how to choose or change CRAN mirror in R or RStudio

Usually R/Rstudio shows list of CRAN mirror at first glance, while installing packages. Often it seems good to select CRAN mirror interactively. For instance, I had chosen a mirror but I have to change due to certain problems in which case the below command works better.

install.packages("your_favorite_package", chooseCRANmirror())

RStudio for Ubuntu 12.04

There are many editors to R, like JGR, Emacs + ESS, Rattle (for data mining), Tinn-R, but I personally like RStudio for many reasons. I shall explain about how to install R and RStudio in Ubuntu 12.04 here:

1. First obtain a copy of RStudio from here.
2. go to update manager -> settings -> add below apt sources:

        deb http:///bin/linux/ubuntu precise/
        if you are in India; possibly your favorite CRAN mirror can be: http://ftp.iitm.ac.in/cran/; for more mirror, you can peep in to this page.
3. go to terminal; alternatively you can also click ctl+alt+t, to trigger gnome-terminal; then add apt-key with the help of below commands;

gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9  

and  

gpg -a --export E084DAB9 | sudo apt-key add -

now update your software sources (respective repositories) with below commands;

sudo apt-get update 

sudo apt-get install r-base

now after installation - install your previously downloaded RStudio package with the

help of software center 

That's all now your RStudio (much coveted IDE for R) is ready. 

Sastri Indo-Canadian Institute & Pondichery University Conference (నా పాండిచేరి యాత్ర)

I started from Pune on 09.06.12, morning 05.00 for 'Shatabdi Express'. I reached Hyderabad at 02.00PM. I got to catch Simhapuri Express, which is scheduled at 10.30 to Nellore, from where I need to catch another train (Pinakini) to leave Chennai, and from Chennai I can travel by Bus. This is my not-so-short itenary of tour. I reached Chennai at 01.30AM, and reached Pondichery at 09.00PM by my friends vehicle (Thanks to Vinob for his help).

This is my first visit to Pondichery, the University is very vast in area and also so sylvanic with lush green campus. As I was already in talk to Prof. Venkatramane, I need to report at guest house 2, where my accommodation was arranged. The accommodation was good, not less than my Thailand Hotel. Room number 207 in International Students Hostel was arranged. Vinobe was also stayed that night in the same room. Vinobe left Pondichery early morning 04.00AM. I was not prepared well for the conference, so I strarted preparing PPTs which are necessary for my presentation. I finished PPTs at 07.00AM. There was a boy who brought a cup of coffee to me.

I left to conference hall (near by cultural centre) along with my other few participants. Morning breakfast was arranged to all participants at the same venue. I proceeded to inaugural session in the adjacent conference hall. There was short film on french colonialism and culture in Pondichery. The inaugural session was so vibrant as His Excellency Lt. Governer Honble. Singh was present. He is really nicest personality I every met till now.

My paper was very fist one in the ground floor (parallel session - 1). I proceeded to the session to present my paper. There were very few presenters, my self along with two other Canadians are scheduled to give talks there. I was not so flinching (as usually I used to be) to present a paper, I was so confident about the innovativeness of my research endeavor. In deed, only one or two really understood my concept. Rest was befuddled about the theme. Later (after I listened to other talks) I understood that my paper was so strong about qualntitative methods, usually people find it difficult to pay attention when it is too technical. But I am sure that for participants it is a learning experience.

The participants asked few questions (I was delighted to get such a response, which I was not extepected) about the methodology. I observed that individuals are not so particular about methods, as most of the papers are so plain and subjective in nature, providing room to intutive cogitation. In fact, I had too much expectations on the research content in papers, but I was upset. I know certain researchers in France and Denmark, but they are very much particular about research, which is not the case here. Any way, I really had good experience, I came to know so many things about western research, attitudes and expectations of the individuals. I should admit that it is one of the good experience to me in my research life.

Locate Residuals on Residual Plots in R

ANOVA is one of the best techniques to analyze variance. In R, although we do ANOVA, it is in general linear model that goes inside. In fact, most of the statistics is all about GLM or generalized linear models, ANOVA, MANOVA, Time series, Regression are, in fact, very few that use linear model calculations in R. I mean to say, the underlying mechanism to these techniques is non other than linear models.

In linear models, i.e., when we we do regression, ANOVA or MANOVA, it is all about residuals nothing else. Do you all accept! just finding F value and after p-value is not all about linear models, but real analysis starts when we shift our concentration to residuals (errors).

I like R for many reasons, one of the reasons is being it is better than any other statistical software not only in reliability but also due to its strong visualization stamina. O kay, to cut to short, let me ask you a question, that if I would like to know what is value of residual on residual plot, what do I need to do? In fact, there is a beautiful command to mark a residual value on residual plots in R.

Say for instance, if I am doing some lm calculations in R and I am to plot residuals in graph, that I can do by executing following code:

> plot(fitted.values(linuxglm), residuals(linuxglm), xlab = "Fitted Values", ylab = "Residuals", col = "red"); abline(h=0)

In above code, linuxglm refers to my linear model (I mean fit). Now the plot can look like below:

 I am not very much satisfied about visualization, I mean, I would like to mine little more about the data, if I have values along with points ( at the same time I don't like to know the values for each and every point, which makes the graph little more messy). As far as this plot is concerned there are two points one at the middle top of the graph, the other at right bottom (perhaps I can call them outliars). I just would like to know about residual values of these points. Now, it can be know by executing following line:

> identify(fitted.values(linuxglm), residuals(linuxglm), xlab = "Fitted Values", ylab = "Residuals", col = "red"); abline(h=0)
 
the moment we press return the R engine becomes busy, after some time, it show as message as locator active (Escape to finish) in R Studio. Don't press esc, first choose to which points you would like to know residual values, then press esc botton. The result will be in terminal:
 
[1]   4 492
 
and teh resultant graph will be:
 


  
 
 

Pie Plots in R

R is very efficient, when it comes to plots; pie plots often useful in describing population characteristics. Suppose if we are to plot pie diagram (plot), say for following data

Female        Male
   85             161

the following can be the code:

> x = c(85, 161)

    
The above code could produce a dataframe with two values with a single row. The data frame can be seen by mearly executing;
  
> x
[1]  85 161
Now we can think of ploting a pie. The diagram should also possess the following;
1) headding
2) labels with respective percentages
3) a legends which must denote same colors as in pie diagram
The following code could prepare labels with respective percentages:
   
> percent = x/sum(x)*100
> labels = c("Female", "Male")
> pielabels = paste(labels, percent, "%")
Now we need to think about the main plot with a headding "Gender". The following command could produce the final plot:
> pie(x, labels = pielabels, col=brewer.pal(7, "Set2"), main="Gender", cex = 0.75,cex.main=0.75); legend("bottomright", legend = c("female", "male"), fill=brewer.pal(7,"Set2"), cex=0.75)

The resultant plot can look like this.
 


 



 




 

 
 

My Bangkok Visit

Recently I had visited Bangkok to attend a conference. I felt its helpful to share my experiences which might be helpful for others.
I am to present a paper on IRCMET conference hosted by Choice Group of Colleges and Global Open University together. I started from Pune on 29th February and reached Mumbai early hours of 01, March to catch an Indigo flight at 06.30 AM. I felt little boredom at airport as was to wait almost 6 hours there. I struck up with small problem as I was asked to show evidence whether my stay got reserved or not. This was new to me as I did not met with such instance before. Finally I somehow convinced the counter-executive at booking by showing conference itenary. Finally I got the boarding pass. Boarded the flight at 06.30 and flight was on schedule.
Reached Bangkok at 01.30 by now I was so frustrated and very badly in need of food (Now-a-days food was stopped in almost all airline from India). There was a long que at immigration check, it took almost 2 hours to get out of it. I collected baggage, lying there as left overs. I wandered here and there for currency exchange. Finally got some exchange kiosk and got approximately 800 Bahts worth of Indian Rupees. I straight proceeded to gate number 12 (as I was instructed in India), there I found a man called Tim, the guide and also very interesting person. All those who travelled with me were there waiting for remaining people. It was almost 03.00PM. In deed, we are suppose to go to hotel and after refreshments follows city tour. But due to delay and skewed planning of travel agent, we were put under city tour first. Of course the city tour is all about Golden Buddha temple visit and followed by a dinner.

The Golden Buddha Temple is must for visit in Bangkok. The great saint/Rishi/god who preached non-violence and compassion was almost forgotten in India. I was so happy to see my forefathers (budda) are being worshipped in a country like Thailand. In fact, thi people love Indian culture most importantly they love Indian movies. I observed this while I was in a cultural show, which is mind blowing at next day.

It was almost 9.00pm we came to 'The First Hotel'. The hotel is well situated in the centre of the city, but little far from 'Suwarna Bhoomi' airport. I was almost exhausted and badly in need of sleep. I had a good sleep at that night.

Bitdefender for Ubuntu

There are so many third party anti-virus applications to Ubuntu. One of the best anti-virus application is Bitdefender. Check Bitdefender for more information.
Bit defender is one of the efficient and effective virus scanner for operating systems like Ubuntu. For home/personal use you can get one year free license, whereby it is possible to update freely for one year. For free license refer here. Just go to the site and register to get one year free license.

Installation:
there are two types of installations one is from the terminal (hassle free); and the other is manual i.e., from the Bitdefender website; go to the Bitdefender portal and download .DEB file from Bitdefender for Unices, just install it by Debian Package manager.

And from the terminal,
first; go to System > Administration > Software Sources; there under other software (second button) add the following line;

deb http://download.bitdefender.com/repos/deb/ bitdefender non-free

then reload software sources; after update there will be a message, don't worry.
go to terminal; Applications > Accessories > terminal; and then type or paste the following lines, to add gpg key;

wget http://download.bitdefender.com/repos/deb/bd.key.asc
sudo apt-key add bd.key.asc

then, again update (but this time from the command-line, by typing or pasting following line;
sudo apt-get update

after the update, type the following command-line;
sudo apt-get install bitdefender-scanner-gui

that's all it will install two files namely; bitdefender-scanner bitdefender-scanner-gui; the file size will be some where around 35MB.

Offline dictioneries in Ubuntu

Unfortunately by default Ubuntu comes with On-line dictionaries. But there is every possibility for either installing off-line dictionaries or making the local dictionaries working off-line.

One of the best efforts to install Off-line dictionary is installing artha. It is basically an Off-line thesaurus based on Wordnet.
just simply type the following line in terminal:

sudo apt-get install artha

the beauty of this is that you can select any word on your desktop and simply press ctl+alt+w thats all artha pop up and gives you required meaning with other stuff.

But, for those who have emotional attachment towards GNOME, you can configure Dictionary server.
the following is the way to installing local dictionary server, with required dictionaries and thesaurus;

sudo apt-get install dictd dictzip dict-wn dict-foldoc dict-jargon dict-gcide dict-moby-thesaurus

and configure the Dictionary in following way;

go to ............ Applications > Office > Dictionary > Edit > Preferences

add a new source using 127.0.0.1 as Hostname and select it as your default source