general linear models vs. generalized linear models

General linear models vs. generalized linear models

• Wikipedia: Comparison of general and generalized linear models (modified)

	General linear model	Generalized linear model
Typical estimation method	Least squares, best linear unbiased prediction	Maximum likelihood or Bayesian
Special cases	ANOVA, ANCOVA, MANOVA, MANCOVA, ordinary linear regression, mixed model, t-test, F-test	linear regression, logistic regression, Poisson regression, gamma regression
Function in R	lm()	glm()
Function in Matlab	mvregress()	glmfit()
Procedure in SAS	PROC GLM, PROC MIXED	PROC GENMOD (PROC LOGISTIC for logistic regression only), PROC GLIMMIX Comparing the MIXED and GLIMMIX
Command in Stata	regress	Stata Generalized linear models
Function in Mathematica	LinearModelFit	GeneralizedLinearModelFit
Command in EViews	ls

• Generalized linear models have the flexiblility for response variables that have other than a normal distribution. If a generalized linear model uses an identity link function and a normal family distribution, then this model is equivalent to a general linear model.
• Generalized linear mixed models have the flexibility to model random effects and correlated errors for nonmormal data.

non-probability sample

Non-Probability Sample


Definition

• Sage Publication (pdf)
• Wikipedia
• Laerd

Reflection

• Wretman (2010) Reflections on Probability vs Nonprobability Sampling (pdf)
• Doherty   (1994) Probability versus Non-Probability Sampling in Sample Surveys (pdf)

Video

• Washington Statistical Society Mini-Conference Presentations (9/9/2015, Vimeo)

All-cause mortality was increasing among US middle age whites

All-cause mortality was increasing among US middle age Whites


Title: Rising morbidity and mortality in midlife among white non-Hispanic Americans in the 21st century
Authors: Anne Case and Angus Deaton
Abstract: This paper documents a marked increase in the all-cause mortality of middle-aged white non-Hispanic men and women in the United States between 1999 and 2013. This change reversed decades of progress in mortality and was unique to the United States; no other rich country saw a similar turnaround. The midlife mortality reversal was confined to white non-Hispanics; black non-Hispanics and Hispanics at midlife, and those aged 65 and above in every racial and ethnic group, continued to see mortality rates fall. This increase for whites was largely accounted for by increasing death rates from drug and alcohol poisonings, suicide, and chronic liver diseases and cirrhosis. Although all education groups saw increases in mortality from suicide and poisonings, and an overall increase in external cause mortality, thosewith less education saw the most marked increases. Rising midlife mortality rates of white non-Hispanics were paralleled by increases in midlife morbidity. Self-reported declines in health, mental health, and ability to conduct activities of daily living, and increases in chronic pain and inability to work, as well as clinically measured deteriorations in liver function, all point to growing distress in this population. We comment on potential economic causes and consequences of this deterioration. Full text: PNAS

Related articles:

• Epimonitor: All Cause mortality rate takes surprising upward turn for middle age Whites in the US
• The Washington Post: Prestigious medical journals rejected stunning study on deaths among middle-aged whites
• The New York Times: Death rates rising for middle-aged White Americans, study finds
• Vox: A big study found spiking death rates for middle-aged white Americans. Critics say it's more complicated
• New York: Anne Case, Co-author of the White Mortality Paper, Responds to a New Critique of Its Approach to Gender (Updated)

How to recover a lost partition of a hard drive

How to recover a lost partition of a hard drive


There are two major reasons you might not see the drive letter of your computer: the logic drive letter lose  or partition table corrupted.

Try these steps first:

• Go to the 'cmd' window by holding the "Windows" key and press the "R" key
• Type and run 'diskmgmt.msc'
• "Disk Management" will be shown.
• If see a partition without a drive letter then right-click on it
• Select "Change Drive Letter and Paths..." 
• Click on "Add" button
• Select the drive letter and Click on OK.

If you cannot see the partition without a drive letter, the partition table may be corrupt, try the TestDisk after the original instruction here or abstracted steps below:

• Download the TestDisk
• Unzip and save it on the USB drive
• Run "testdisk_win"
• At the first window, select “No Log” and press the key
• Select which drive to analyse, choose “Proceed” and press key
• Select partition type (select "Intel" if it’s a PC) then press key
• Select “Analyse” then press key
• Select “Quick Search” at the next screen, then press  key
• Press key, if the partitions were created under Vista – press key if not.
• TestDisk should say “Structure OK”. If so, press key 
• Select “Write” and press key and press   key to confirm.
• "ok" to reboot the compute, press key
• Now, close TestDisk and RESTART the computer.

R documentation and Learning Resources

R! documentation and Learning Resources

• Blog: R commands and keyboard shortcuts
• Blog: R! Books
• Rdocumentation of Datacamp is a tool that helps you easily find and browse the documentation of all current and some past packages on CRAN.
• CRAN Task View organizes the packages into different groups such as Graphics, Survival Analysis, etc.
• R_note -- The Exploration of Statistical Software R
• R! Tips
• Resources of learning R, UCLA
• Quick-R is to help you quickly access this language in your work
• William N Venables, David M Smith, and the R Core Team. An Introduction to R (on-going updated)
• Daniel Navarro (2015). Learning Statistics with R
• Sharon Machlis (2013).Beginner's guide to R: Introduction (pdf), and (2016) Advanced Beginner's Guide to R (pdf). 60+ R resources to improve your data skills
• Emmanuel Paradis. R for Beginners
• Paul Johnson (2014). R Tips
• ZevRoss. Beautiful plotting in R: A ggplot2 Cheatsheet (pdf)
• Alboukadel Kassambara. ggplot2: Guide to Create Beautiful Graphics in R
• Jay Kerns (2010). Introduction to Probability and Statistics Using R
• Julian Faraway (2002) Practical Regression and Anova using R
• Benjamin Yakir (2011). Introduction to Statistical Thinking (With R, Without Calculus)
• Paul Helwson (2009). Multivariate Statistics with R
• Hadley Wickham. ggplot2 documents (Plotly ggplot2 docs with Plotly's open-source ggplotly converter)
• RStudio. Cheat Sheets, Webinars and Video
• Norman Matloff (2009). The Art of R Programming
• Gillespie (2016). Efficient R Programming
• Scherer: A ggplot2 tutorial for beautiful plotting in R
• Thomas Lumley (2011). Course Notes - Complex sampling and R use survey package. withReplicates {survey} computes variances by replicate weighting.
• Anthony Damico:  analyze survey data for free, unlocked public-use data sets, twotorials
• Aedin Culhane: Introduction to R, CDC, June 25-28th 2012 (it's one of my favorite training courses attended). Here is a Amazon list of R TextBooks by Aedin.
• Yiling Cheng (2016). Doing Basic Calculus using R! etc.
• Verzani (2002). simpleR - Using R for Introductory Statistics (pdf)
• Patrick Burns (1998). S Poetry
• Zhou (2010). Fun with the R Grid Package
• Torre-Reyna: R: A language and Environment for statistical computing (Full manual)
• Getting started in R~Stata notes on exploring data (pdf)
• Muenchen: R for SAS and SPSS Users (pdf). the list of SAS equivalent packages.
• Maindonald: Using R for Data Analysis and Graphics (pdf)
• Data Analysts Captivated by R’s Power from the New York Times.
• Kadane. Principles of Uncertainty (pdf)
• Kerns. Introduction to Probability and Statistics Using R (pdf)
• Gaston. Handling and Processing Strings in R
• Frank Harrell. Regression Modeling Strategies and the rms Package
• Effective Graphs with Microsoft R Open
• Roger Peng and Jeff Leek. Git and GitHub videos for beginners
• Roger Peng etc. Course materials of Data Science Specialization
• Kevin Markham (2014) Hands-on dplyr tutorial for faster data manipulation in R
• Hopper (2016).The Excel User's introduction to R!
• Rickert. Getting Started with Markov Chains
• Wikibook. R programming
• Walker (2014). International population pyramids with ggplot2
• Demos: A Language, not a Letter: Learning Statistics in R

Running and longevity

Running and Longevity

News

• DialyMail: Stop that binge jogging
• Long term study found slow joggers had the lowest rates of death
• Strenuous joggers were as likely to die as sedentary non-joggers
• Going jogging three times a week for no more than 2.4 hours is optimal
• Pace of the slow joggers corresponds to vigorous exercise and strenuous jogging corresponds to very vigorous exercise, researchers qualified
• ScienceDaily: Light jogging may be most optimal for longevity

Guidelines and Researches

• CDC: How much physical activity do you need?
• Schnohr (2015). Dose of Jogging and Long-Term Mortality: the Copenhagen City Heart Study.
• Mittlema (1993). Triggering of Acute Myocardial Infarction by Heavy Physical Exertion
• Cheng (2000): Physical activity and self-reported, physician-diagnosed osteoarthritis: is physical activity a risk factor?
• Cheng (2012): Prevalence of diagnosed arthritis and arthritis-attributable activity limitation among adults with and without diagnosed diabetes

The Roseto Mystery

The Roseto Mystery

A few days ago, I read a book, Outliers (2008), by Malcolm Gladwell. The book is quite insightful. All children, parents, grandparents (ha-ha), and educators can be benefited from reading it. The introduction chapter is about the Rosetans who living in the Pennsylvannia, which is more related to my work. The Rosetans are outliers who are eating a big portion of saturated fat including lard with some other bad habits such as smoking. However, “THESE PEOPLE WERE DYING OF OLD AGE. THAT’S IT.” The Gladwell showed some evidences of importance of a low stressful lifestyle, good supports from family, friends, and communities for the health and longevity.

Here are a few more stories related to the Rosetans who living in Roseto, Pennsylvania:

• The first Chapter of Outliers
• Joe (2011). The Roseto Effect – The Valley of the Roses
• Steward Wolf (1993). The Power of Clan
• Cal Gutkin (2009). Outliers: extended families, better health outcomes
• Rock Positano (2011). The Mystery of the Rosetan People

Cancer isn’t just bad luck

Cancer isn’t just bad luck
By Thomas Lumley

"
From Stuff, "Bad luck is responsible for two-thirds of adult cancer while the remaining cases are due to environmental risk factors and inherited genes, researchers from the Johns Hopkins Kimmel Cancer Center found."

...

So, in summary: the “two-thirds of cancers explained” is Just Wrong. Doing a mathematically correct calculation gives about one third. Doing a calculation that’s actually relevant to cancer in the population gives even smaller values. (update) That’s not to say that DNA replication errors are unimportant — the paper makes it clear that they are important.
"

The fulltext: Cancer isn't just bad luck

Chris Botti in Boston, 2008.

Beautiful Sound of Trumpet by Chris Botti in Boston, 2008

Christopher Stephen "Chris" Botti (born October 12, 1962), is an Italian-American trumpeter and composer. Wikipedia - Chris Botti

Pavel Levin, "Chris Botti in Boston features trumpeter Chris Botti along with a bevy of name artists performing live with the Boston Pops Orchestra at Symphony Hall in 2008. Fully documented as a concert film and album, the night is an intimate and soulful birds-eye view of the supple-toned trumpeter who has grown into his role as a virtuoso since his time backing up Sting ..." 

How to become a data scientist

How to Become a Data Scientist in 8 Easy Steps (JPG Figure)
Source: DataCamp Blog

Data Cleaning is a critical part of the Data Science process

Data Cleaning is a critical part of the Data Science process
by David Smith

A New York Times article yesterday discovers the 80-20 rule: that 80% of a typical data science project is sourcing cleaning and preparing the data, while the remaining 20% is actual data analysis. The article gives short shrift to this important task by calling it "janitorial work", but whether you call it data munging, data wrangling or anything else, it's a critical part of the data science. I'm in agreement with Jeffrey Heer, professor of computer science at the University of Washington and a co-founder of Trifacta, who is quoted in the article saying,

     “It’s an absolute myth that you can send an algorithm over raw data and have insights pop up.”

As an illustration of this point, check out the essay by Julia Evans, Machine learning isn't Kaggle competitions (hat tip: Drew Conway). A Kaggle competion typically presents a nice, clean, regularized data set to the competitors, but this isn't representative of the real-world process of making predictions from data. As Julia points out:

     Cleaning up data to the point where you can work with it is a huge amount of work. If you’re trying to reconcile a lot of sources of data that you don’t control like in this flight search example, it can take 80% of your time.

While there are projects underway to help automate the data cleaning process and reduce the time it takes, the task of automation is made difficult by the fact that the process is as much art as science, and no two data preparation tasks are the same. That's why flexible, high-level langauages like R are a key part of the process. As Mitchell Sanders notes in a Tech Republic article,

     Data science requires a difficult blend of domain knowledge, math and statistics expertise, and code hacking skills. In particular, he suggests that expert knowledge of tools like R and SAS are critical. "If you can't use the tools, you can't analyze the data."

This is a critical step to gaining any kind of insight from data, which is why data scientists still command premium salaries today, according to data from Indeed.com.

How to generate a color-coded/formatted Word file of R programs in RStudio

RMarkdown: Generate a color-coded/formatted Word file of R! using RStudio

R Markdown — Dynamic Documents for R

• Yihui Xie on RMarkdown v2 (with vide)
• Basics
• Documentation 

Here is my approaches/cheat-sheet/template:

If have codes like these:
 

     library("xxx")    

     summary (usmort)

 

Method I (using RStudio)

• Start a new R markdown file (File -> New File-> R Markdown).
• Delete all the codes that RStudio generated in that new file.
• Copy and paste the R codes above into the new file
• Add global option (between --- and ---), chunk head (```) with option (eval=FALSE), and chunk tail (```) :

    ---

    output:

    word_document:

    highlight: pygments

    ---

    ```{r,eval=FALSE}

                 ... your codes here ...

    ``` 

• Click the ‘knit word’ button, you get a Word file with codes show like below: 

     library("xxx")
     summary (usmort)

 

Method II (with or without using RStudio)

• add chunk head (```{r,eval=FALSE}) and tail (``` ) and wrap the whole codes, and save the file ("xxx_r_code.R").
• run the R commands in console window:

       >library(rmarkdown)

    >render("xxx_r_codes.R",word_document(highlight="pygments"))

Poison Ivy

Poison Ivy
 
Diagnose Poison Ivy Rash

• Poison ivy: Signs and symptoms
• How Poison Ivy Works

Treat Poison Ivy Rash

• Tips for treating and preventing
• My experience:
• Wash the affect area with Dawn dish soap.
• Use Caladryl (Pink Lotion, Clear Lotion) to treat itching.
• Wash contacted clothes daily.
• Wash affect itching area with cold water than hot water (don't burn yourself).
• Get helps from your family doctor. This may be the better way to saves you money and time.

Identify Poison Ivy

• Rhymes of identifying poison ivy (YouTube):
• Leave of three, beware of thee.
• Longer middle stem, stay away from them.
• Side leafs like mittens, itch like the dickens.
• Red leaves in spring, it’s a dangerous thing.
• Hairy vine, no friend of mine.
• Berries white, danger in sight.
• YouTube video:
  
  • How to Get Rid of Poison Ivy
  • Poison Oak/Ivy - A Survival Guide

 
Remove Poison Ivy

• Tips for removing poison ivy safely
• What is the best way to get rid of poison ivy? 
• Controlling Poison Ivy in the Landscape
• Herbicides
• Ortho: Brush-B-Gone
• RoundUp: How To Kill Poison Ivy
• Brush-Be-Gone Vs. Roundup
• 7 Ways to kill poison ivy without using Roundup
• Vinegar for poison ivy
• 1 cup of salt
• 8 drops of liquid detergent
• 1 gallon of vinegar.
• combine the salt and vinegar in a pan heat it up to dissolve the salt pour into a spray bottle and kill away. Warning this will kill all plants (?) that you spray so don't spray anything important good luck.

 
 

Simpson's Paradox

Simpson's Paradox

Simpson's paradox (or the Yule-Simpson effect) is an unexpected result in which a association/correlation of  x and y in different groups disappears or reverses when the groups are combined.

• Wikipedia. Simpson's paradox and other Statistical paradoxes
• Simpson's Paradox
• Bob Horton (2015). Fun with Simpson's Paradox: Simulating Confounders
• Natloff (2014). Simpson's paradox is back
• Revolutions (2013). An interactive tool to explain Simpson's Paradox
• Tu (2008). Simpson's Paradox, Lord's Paradox, and Suppression Effects are the same phenomenon – the reversal paradox
• Baker (2001). Good for women, good for men, bad for people
• Taylor (2014). Simpson’s Paradox: A Data Set and Discrimination Case Study Exercise

uncertainty

When scientist and policy maker plus uncertainty

• Schmitt (2014). "The Pitfall of Uncertainty" (The Scientist)
• The Scientist published an opinion article about what uncertainty means to scientisit and policy maker. "...they must be crystal clear: science is about discovery and (decreasing) uncertainty, policymaking is about achieving consensus (if not certainty). Together, scientists and policymakers alike must strive to make responsible decisions for the benefit of society."
• NPR (2017). Alan Alda's Experiment: Helping Scientists Learn To Talk To The Rest Of Us
• Wikipedia:
  • Truth
  • Certainty
  • Wikiquote. Truth
• A intuitive figure of true vs truth of Starecat.com: This is TRUE, this is TRUE, this is TRUTH. I like this figure indeed. However, 'True' is a an adjective; 'truth' is a noun. I would like to change this into "This is TURE, this is TURE, this is THE THUTH" or "This is a FACT, this is a FACT, this is THE TRUTH"

    

Proposed Revisions to the Common Rule

Proposed Revisions to the Common Rule for the Protection of Human Subjects in the Behavioral and Social Sciences

It examines how to update human subjects protections regulations so that they effectively respond to current research contexts and methods. With a specific focus on social and behavioral sciences, this consensus report aims to address the dramatic alterations in the research landscapes that institutional review boards (IRBs) have come to inhabit during the past 40 years. The report aims to balance respect for the individual persons whose consent to participate makes research possible and respect for the social benefits that productive research communities make possible... (full text)