Calculate Normal Distribution Using R

#:************************************************************************
#: Author: Alex Madriaga
#: Date: 06/06/2013
#: Time: 11:11:11
#: File: dnorm.r
#: Status: Working using the data in C:/workspace/ecdf/d176m.csv
#: Version: 0.98
#: Description: Latest version of the normal distribution for R scripting
#: Revision history:
#:
#:************************************************************************

# load the csv file, ignoring the header
mainDir <- "C:/workspace/ecdf"
setwd(mainDir)
mainDir <- getwd()
rawCSVFile <- "d176m.csv"
rsm <- read.csv(rawCSVFile,header=TRUE)

# check structure and convert vector values to numeric
str(rsm)
rsm$HEAP <- as.numeric(rsm$HEAP)

# calculate the mean and standard deviation
mean <- mean(rsm$HEAP)
sd <- sd(rsm$HEAP)

#plotting the density function of a normal distribution
heap <- rsm$HEAP

# plot the data
plot(heap, dnorm(heap, mean, sd), type="l")

Bash Script using Flags and Options

#!/bin/bash

#:########################################################################
#: The option-string
#:
#: The option-string tells getopts which options to expect and which of them
#: must have an argument. The syntax is very simple . every option character
#: is simply named as is, this example-string would tell getopts to look
#: for -e, -d and -m:
#:
#: getopts fAx VARNAME
#:
#: When you want getopts to expect an argument for an option, just place
#: a : (colon) after the proper option flag. If you want -e to expect an
#: argument (i.e. to become -e ENVIRONMENT ) just do:
#:
#: getopts ed:m VARNAME
#:
#:########################################################################

function checkParam()
{

  # if no paramter was supplied, show usage
  text="Usage: $0 -e ENVIRONMENT -d DATE_FILENAME.TXT -m METRIC_FILENAME.TXT" 
  [ $# -eq 0 ] && { echo "$text"; exit 1; }

  # parse the flags and options
  while getopts "e:d:m:" opt; do
    case $opt in
      e)
        echo "-$opt $OPTARG $OPTIND $OPTERR was triggered!" >&2
        env=$OPTARG
        ;;
      d)
     echo "-$opt $OPTARG $OPTIND $OPTERR was triggered!" >&2
        fdate=$OPTARG
        ;;
      m)
     echo "-$opt $OPTARG $OPTIND $OPTERR was triggered!" >&2
        fmetric=$OPTARG
        ;;
      *)
        echo "Invalid option: -$opt $OPTARG $OPTIND $OPTERR" >&2
        ;;
    esac
  done

  # show the flags and options that were parsed
  echo "Environment: [$env]"
  echo "Date filename: [$fdate]"
  echo "Metric filename: [$fmetric]"

  # Check the given files exist #
  [ ! -f "$fdate" ] && { echo "Error: $fdate file not found."; exit 2; }
  [ ! -f "$fmetric" ] && { echo "Error: $fmetric file not found."; exit 2; }

  # Check the given files are not empty #
  [ ! -s "$fdate" ] && { echo "Error: $fdate file is empty."; exit 2; }
  [ ! -s "$fmetric" ] && { echo "Error: $fmetric file is empty."; exit 2; }
}

#: ## [ MAIN ] ###
checkParam $@

Enable SSH on WD MyBook

Link

       

To login use /UI/login, example: http://192.168.1.150/UI/login

Open Mybook UI go to Import / Export Current 
Configuration (under Utilities)

Export the current configuration, open the file, update: 
ssh_enable="disabled", to ssh_enable="enabled", import 
the configuration file

After reboot SSH is enabled.

Default is root / welc0me as per other WD devices.

Change root password right away.


If you want to enable SFTP, modify /etc/ssh/sshd_config 
and add the user who needs to access through SFTP. 
For example: “AllowUsers root malex” enables the user 
“malex” to connect to the device by both ssh and sftp.

Twonky is available at port 9000, example: http://192.168.1.150:9000
FTP when enabled is available using the user you created.

Load a CSV file and plot the normal distribution curve

# load the csv file, ignoring the header
mainDir <- "C:/workspace/ecdf"
setwd(mainDir)
mainDir <- getwd()
rawCSVFile <- "heap.csv"
rsm <- read.csv(rawCSVFile,header=TRUE)

# check structure and convert vector values to numeric
str(rsm)
rsm$HEAP <- as.numeric(rsm$HEAP)

# calculate the mean and standard deviation
mean <- mean(rsm$HEAP)
sd <- sd(rsm$HEAP)

#plotting the density function of a normal distribution
heap <- rsm$HEAP

# plot the data
plot(heap, dnorm(heap, mean, sd), type="l")

Theory on Cross- over ratio of powers

My theory is that the ratio of the electrical power with respect to the software power is constant .Thus, the economics of cost is easily computed in terms of monies.

Theory on Electrical power vs. Soft power

The power consumed by the server and all the devices connected to the whole application platform expressed in electrical Watts is proportional to the power generated by the server expressed in S-WATTS or soft watts or watts subscript small s. This power must also be equal to the power consumed by the users at the receiving end.

Theory on Powet factor correction on JVM

Having understood that the JVM is indeed comparable with an AC generator, we are now able to calculate the power factor and efficiency of the server using the vast theories and formula in electrical engineering.

Calculation of R on a system

To measure the R constant of the system, we need to conduct several heap samples at varying number of users. Then calculate the R constant based on the empirical data. That is, R is equal to HSU divided by users.

Potential difference in JVM using VIR model

The potential difference across a server can be calculated using the electrical engineering equation V = IR. The equivalent equation on the JVM is T=HU.

JVM Capacity Model using I^2 R

The electrical engineering equation of power I^2R can be used to calculate the capacity of the server expressed in H^2U, where H is heap memory used in MB and U can be the number of users connected to the servers.Thus Capacity (C) = H^2U. The unit is HSU (Heap Squared Users.

Phase angle differences between the heap and the CPU utilization

When applications are loaded, the tendency of the server is to react accordingly. The question now arise as to which one will react first to the load- the HEAP memory or the CPU usage. My theory is that one of them will be the first and the second will be having a phase angle difference with the other. The sample calculation will be shown later.

Theory on Harmonics In Java-based applications

I discovered yesterday that the heap memory usage of a certain java application  behaves like a triangular waveform and thus can be calculated using RMS values to determine the HEAP at any point in time during runtime. Thus, the frequency of the heap cycles can be described in hertz and heap as Heap-RMS. Having known this behavior, the detection of heap memory leak is possible. Consider a server at no load for a significant amount of time is tested for memory leak. The integral of all d(h)/dt from h(i) to h(f) from t(0) to t(f) must be equal to zero. Even a small delta will indicate a creaping memory leak The calculation will be shown later.

Hosting your code in sourceforge.net

Steps on hosting your code at sourceforge.net

1. Create an account here: https://sourceforge.net/user/registration
2. Use sftp, WinSCP or Altap Salamander
   Using sftp below:

    linuxuser@localhost:~$ sftp your_user_name@frs.sourceforge.net
    The authenticity of host 'frs.sourceforge.net (216.34.181.57)' can't be established.
    RSA key fingerprint is 06:b0:b0:ca:b0:ca:ca:ca:ba:00:b0:b0:ca:b0:b0:ca.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'frs.sourceforge.net,216.34.181.57' (RSA) to the list of known hosts.
    your_user_name@frs.sourceforge.net's password:
    Connected to frs.sourceforge.net.
    sftp> ls
    sftp> exit
    linuxuser@localhost:~$
3. My account is here: http://azmadriaga.users.sourceforge.net/

Some useful Windows and Linux Programs

Navigate iPad filesystem

i-Funbox

Home: http://www.i-funbox.com/


Description: No Jailbreak SSH, Manage files on your iPhone/iPad just like Windows Explorer on your PC, but in a more robust and friendly way

Simple yet visually awesome code editor and highlighter 

Context 

Home: http://www.contexteditor.org/ 
Highlighters: http://www.contexteditor.org/highlighters/ 

Description: ConTEXT is a small, fast and powerful freeware text editor, developed to serve as a secondary tool for software developers.

Securely delete file using command-line-interface (CLI) in Windows  

SDelete 

Home: http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx 

Description: SDelete implements the Department of Defense clearing and sanitizing standard DOD 5220.22-M, to give you confidence that once deleted with SDelete, your file data is gone forever. Note that SDelete securely deletes file data, but not file names located in free disk space.

Securely delete file using command-line-interface (CLI)  in Linux

SRM: Secure-Delete 

Home: http://techthrob.com/2009/03/02/howto-delete-files-permanently-and-securely-in-linux/
Install: apt-get install secure-delete
Description: This tool is basically a more advanced version of the “shred” command. Instead of just overwriting your files with random data, it uses a special process – a combination of random data, zeros, and special values developed by cryptographer Peter Gutmann – to really, really make sure your files are irrecoverable. It will assign a random value for the filename, hiding that key piece of evidence.

Recover deleted files in Windows/Linux/Mac

TestDisk

Home: http://www.cgsecurity.org/wiki/TestDisk
Description:TestDisk is powerful free data recovery software! It was primarily designed to help recover lost partitions and/or make non-booting disks bootable again when these symptoms are caused by faulty software, certain types of viruses or human error (such as accidentally deleting a Partition Table). Partition table recovery using TestDisk is really easy.

Split and merge pdf documents, it's free, open source and platform independent

PDFsam

Home: http://www.pdfsam.org/
Description:PDFsam basic is a simple, platform independent software designed to split and merge pdf files. It’s stable, completely free and It should cover most of your needs.

A UI version of diff in Windows

WinMerge

Home: http://winmerge.org/
Description:WinMerge is an Open Source differencing and merging tool for Windows. WinMerge can compare both folders and files, presenting differences in a visual text format that is easy to understand and handle.

Password keeper

KeePass

Home: http://keepass.info/
Description:KeePass is a free open source password manager, which helps you to manage your passwords in a secure way. You can put all your passwords in one database, which is locked with one master key or a key file.

Passwork keeper for Android phones

Pocket

Home: http://www.appbrain.com/app/pocket/com.citc.wallet
Description:Pocket allows you to safely store all your sensitive data such as bank account details and passwords on your phone. Pocket is also useful for remembering all those bits and pieces of information in one place from frequent flier numbers to contact lens prescriptions.

Fast Image Resizer

adionSoft Fast Image Resizer

Home: http://adionsoft.net/fastimageresize/
Description:Resize images to any size quickly and in high quality. Can read JPG, BMP, GIF, PNG, TIFF and HD Photo (.wdp, .hdp) files. Writes JPG, BMP or PNG files. Compatible with Windows XP, Vista, Windows 7, Windows 8 and Mac OS X 10.6

Calculating ECDFS using R

        # store this current column to a variable
        pcs <-rsm[[colnam]]

#-- additional implementation
#        pcs2 <- pcs[!duplicated(pcs)]
#        pcs <- pcs2

#        pcs3 <- pcs[ pcs != 0 ]
#        pcs <- pcs3

#--
     # sort the data and store the sorted value to a new variable
        sortpcs <- sort(pcs)

        # calculate the CDF score based on the number of rows
        ecdfpcs <- (1:length(sortpcs))/length(sortpcs)

        # create a data frame from the data
        svrCdf <- data.frame(sortpcs,ecdfpcs)

#-- additonal implementaion
        stop("Message")
        warning("Message")
#--

# hard code the specific directory
        dir <- "c:/amadriaga/a_r"
        setwd(dir)
        dir <- getwd()
        print(dir)
        svrCdfCsvFile <- sprintf("%s/ecdfs_%s_norm.csv",dir,colnam)
        write.csv(svrCdf, file=svrCdfCsvFile, row.names=FALSE)
        print("Successfully completed R script.")

Adding Legend in R using Column names

# add legend to the graph
 legend("topright", legend = c(headerNames[2:length(headerNames)]),
        col=2:length(rsm), pch=20, lty=1,
        lwd=0.75, bty="n", cex=0.75)

Adding Gridlines in R Manually using abline()

  # vertical grid lines
  xMin <- 0
  xMax <- 10
  xStep <- 1
  for (x in seq(xMin, xMax, xStep)){
   print (x)
   abline(v=(seq(x,100,25)), col=195, lty="dotted")
  }
  # highlight on x=4 and x=5
  abline(v=(seq(4,100,25)), col="red", lty=1)
  abline(v=(seq(5,100,25)), col="red", lty=1)

  # horizontal grid lines
  yMin <- 0
  yMax <- 1
  yStep <- 0.1
  for (y in seq(yMin, yMax, yStep) ){
   print (y)
   abline(h=(seq(y,100,25)), col=195, lty="dotted")
  }

  # highlight at y=0.8
  abline(h=(seq(0.8,100,25)), col="red", lty=1)

Determine Column Number based on Column Names

    # print the column name
    print ( colnam )

    # print the number of columns, this is not changing
    print ( counter <-length(rsm) )

    # determine the column number based on the column name
    # and store to a variable
    columnNumber <- which( colnames(rsm)==colnam )

    # print the column number
    print ( columnNumber )

Reading CSV and printing the headers

# Read the raw data
rsm <- read.csv("c:/amadriaga/a_r/rawServerMemory.csv")

# Save the column names to array
headerNames <- colnames(rsm)

# Get count of columns
print (length(rsm))

# Loop on column headers and print the header names
for (colnam in names(rsm)[ 1:length(rsm) ] ){
    # print the column name
    print ( colnam )

    # print the number of columns, this is not changing
    print ( counter <-length(rsm) )

    # determine the column number based on the column name
    # and store to a variable
    columnNumber <- which( colnames(rsm)==colnam )

    # print the column number
    print ( columnNumber )
}