STAT 29000

Project 6

due October 31, 2016

Put all of your solutions into one text file for your group. For example, the file for project 6, group 3, should be called: p06g3.Rmd and it should be stored in the folder: /proj/gpproj16/p06g3/ Also include the results of the RMarkdown, i.e., include either the html, pdf, or Word output from your RMarkdown file.

PREAMBLE

During Project 5, question 5, you scraped the data from all of the BillBoard charts. In this project, you can have freedom to explore questions about the data.

To scrape the information from the first Hot 100 chart, you can use XPath. First it is necessary to install XPath.

install.packages("XML",repos="http://cran.us.r-project.org")
## Installing package into '/Users/mdw/Library/R/3.2/library'
## (as 'lib' is unspecified)
## 
## The downloaded binary packages are in
##  /var/folders/mp/csj2s8nn50s80rx6kjvzh1900000gn/T//Rtmpf0WBue/downloaded_packages
library("XML")

Then it is necessary to parse the XML source.

mydoc <- htmlParse("/Users/mdw/1958-08-09")

Now you are able to make queries about the XML content in a page. For instance,

mysongs <- xpathSApply(mydoc, "//*/div[@class='chart-row__title']/h2[@class='chart-row__song']", xmlValue)
mysongs
##   [1] "Poor Little Fool"                                        
##   [2] "Patricia"                                                
##   [3] "Splish Splash"                                           
##   [4] "Hard Headed Woman"                                       
##   [5] "When"                                                    
##   [6] "Rebel-'Rouser"                                           
##   [7] "Yakety Yak"                                              
##   [8] "My True Love"                                            
##   [9] "Willie And The Hand Jive"                                
##  [10] "Fever"                                                   
##  [11] "Ginger Bread"                                            
##  [12] "Just A Dream"                                            
##  [13] "Left Right Out Of Your Heart (Hi Lee Hi Lo Hi Lup Up Up)"
##  [14] "If Dreams Came True"                                     
##  [15] "For Your Precious Love"                                  
##  [16] "One Summer Night"                                        
##  [17] "Endless Sleep"                                           
##  [18] "Little Star"                                             
##  [19] "Everybody Loves A Lover"                                 
##  [20] "Do You Want To Dance"                                    
##  [21] "Guess Things Happen That Way"                            
##  [22] "A Certain Smile"                                         
##  [23] "Western Movies"                                          
##  [24] "The Purple People Eater"                                 
##  [25] "What Am I Living For"                                    
##  [26] "Born Too Late"                                           
##  [27] "Think It Over"                                           
##  [28] "Secretly"                                                
##  [29] "Enchanted Island"                                        
##  [30] "Angel Baby"                                              
##  [31] "Chantilly Lace"                                          
##  [32] "Blue Blue Day"                                           
##  [33] "The Freeze"                                              
##  [34] "Don't Ask Me Why"                                        
##  [35] "Rock-in Robin"                                           
##  [36] "No Chemise, Please"                                      
##  [37] "Moon Talk"                                               
##  [38] "Somebody Touched Me"                                     
##  [39] "That's How Much I Love You"                              
##  [40] "Crazy Eyes For You"                                      
##  [41] "Early In The Morning"                                    
##  [42] "You Cheated"                                             
##  [43] "Come What May"                                           
##  [44] "Jennie Lee"                                              
##  [45] "Kathy-O"                                                 
##  [46] "(It's Been A Long Time) Pretty Baby"                     
##  [47] "I Wonder Why"                                            
##  [48] "Return To Me"                                            
##  [49] "All I Have To Do Is Dream"                               
##  [50] "By The Light Of The Silvery Moon"                        
##  [51] "Baubles, Bangles And Beads"                              
##  [52] "Early In The Morning"                                    
##  [53] "Come Closer To Me (Acercate Mas)"                        
##  [54] "Nel Blu Dipinto Di Blu (Volare)"                         
##  [55] "Let's Go Steady For The Summer"                          
##  [56] "Leroy"                                                   
##  [57] "You Need Hands"                                          
##  [58] "Fool's Paradise"                                         
##  [59] "Young And Warm And Wonderful"                            
##  [60] "Over And Over"                                           
##  [61] "Itchy Twitchy Feeling"                                   
##  [62] "For Your Love"                                           
##  [63] "High School Confidential"                                
##  [64] "Padre"                                                   
##  [65] "You're Making A Mistake"                                 
##  [66] "Delicious!"                                              
##  [67] "Big Man"                                                 
##  [68] "Volare (Nel Blu Dipinto Di Blu)"                         
##  [69] "Op"                                                      
##  [70] "Don't Go Home"                                           
##  [71] "Got A Match?"                                            
##  [72] "Stupid Cupid"                                            
##  [73] "Hey Girl - Hey Boy"                                      
##  [74] "Gotta Have Rain"                                         
##  [75] "Win Your Love For Me"                                    
##  [76] "Midnight"                                                
##  [77] "Happy Years"                                             
##  [78] "Betty Lou Got A New Pair Of Shoes"                       
##  [79] "The Bird On My Head"                                     
##  [80] "Johnny B. Goode"                                         
##  [81] "Beautiful Delilah"                                       
##  [82] "Blip Blop"                                               
##  [83] "Try The Impossible"                                      
##  [84] "Summertime Blues"                                        
##  [85] "Got A Match?"                                            
##  [86] "To Be Loved"                                             
##  [87] "Jealousy"                                                
##  [88] "Just Like In The Movies"                                 
##  [89] "Blue Boy"                                                
##  [90] "Stay"                                                    
##  [91] "The Purple People Eater Meets The Witch Doctor"          
##  [92] "Bird Dog"                                                
##  [93] "Are You Really Mine"                                     
##  [94] "She Was Only Seventeen (He Was One Year More)"           
##  [95] "Little Mary"                                             
##  [96] "Over And Over"                                           
##  [97] "I Believe In You"                                        
##  [98] "Little Serenade"                                         
##  [99] "I'll Get By (As Long As I Have You)"                     
## [100] "Judy"

or like this

myartists <- xpathSApply(mydoc, "//*/div[@class='chart-row__title']/h3[@class='chart-row__artist']|//*/div[@class='chart-row__title']/a[@class='chart-row__artist']", xmlValue)
myartists
##   [1] "\n                                Ricky Nelson\n                            "                               
##   [2] "\n                                Perez Prado And His Orchestra\n                            "              
##   [3] "\n                                Bobby Darin\n                            "                                
##   [4] "\n                                Elvis Presley With The Jordanaires\n                            "         
##   [5] "\n                                Kalin Twins\n                            "                                
##   [6] "\n                                Duane Eddy His Twangy Guitar And The Rebels\n                            "
##   [7] "\n                                The Coasters\n                            "                               
##   [8] "\n                                Jack Scott\n                            "                                 
##   [9] "\n                                The Johnny Otis Show\n                            "                       
##  [10] "\n                                Peggy Lee\n                            "                                  
##  [11] "\n                                Frankie Avalon\n                            "                             
##  [12] "\n                                Jimmy Clanton And His Rockets\n                            "              
##  [13] "\n                                Patti Page\n                            "                                 
##  [14] "\n                                Pat Boone\n                            "                                  
##  [15] "\n                                Jerry Butler and The Impressions\n                            "           
##  [16] "\n                                The Danleers\n                            "                               
##  [17] "\n                                Jody Reynolds\n                            "                              
##  [18] "\n                                The Elegants\n                            "                               
##  [19] "\n                                Doris Day\n                            "                                  
##  [20] "\n                                Bobby Freeman\n                            "                              
##  [21] "\n                                Johnny Cash And The Tennessee Two\n                            "          
##  [22] "\n                                Johnny Mathis\n                            "                              
##  [23] "\n                                The Olympics\n                            "                               
##  [24] "\n                                Sheb Wooley\n                            "                                
##  [25] "\n                                Chuck Willis\n                            "                               
##  [26] "\n                                Poni-Tails\n                            "                                 
##  [27] "\n                                The Crickets\n                            "                               
##  [28] "\n                                Jimmie Rodgers\n                            "                             
##  [29] "\n                                The Four Lads\n                            "                              
##  [30] "\n                                Dean Martin\n                            "                                
##  [31] "\n                                Big Bopper\n                            "                                 
##  [32] "\n                                Don Gibson\n                            "                                 
##  [33] "\n                                Tony And Joe\n                            "                               
##  [34] "\n                                Elvis Presley With The Jordanaires\n                            "         
##  [35] "\n                                Bobby Day\n                            "                                  
##  [36] "\n                                Gerry Granahan\n                            "                             
##  [37] "\n                                Perry Como\n                            "                                 
##  [38] "\n                                Buddy Knox with the Rhythm Orchids\n                            "         
##  [39] "\n                                Pat Boone\n                            "                                  
##  [40] "\n                                Bobby Hamilton\n                            "                             
##  [41] "\n                                Buddy Holly\n                            "                                
##  [42] "\n                                The Slades\n                            "                                 
##  [43] "\n                                Clyde McPhatter\n                            "                            
##  [44] "\n                                Jan & Arnie\n                            "                                
##  [45] "\n                                The Diamonds\n                            "                               
##  [46] "\n                                Gino & Gina\n                            "                                
##  [47] "\n                                Dion & The Belmonts\n                            "                        
##  [48] "\n                                Dean Martin\n                            "                                
##  [49] "\n                                The Everly Brothers\n                            "                        
##  [50] "\n                                Jimmy Bowen with the Rhythm Orchids\n                            "        
##  [51] "\n                                The Kirby Stone Four\n                            "                       
##  [52] "\n                                The Rinky-Dinks\n                            "                            
##  [53] "\n                                Nat King Cole\n                            "                              
##  [54] "\n                                Domenico Modugno\n                            "                           
##  [55] "\n                                The Three G's\n                            "                              
##  [56] "\n                                Jack Scott\n                            "                                 
##  [57] "\n                                Eydie Gorme\n                            "                                
##  [58] "\n                                The Crickets\n                            "                               
##  [59] "\n                                Tony Bennett\n                            "                               
##  [60] "\n                                Bobby Day\n                            "                                  
##  [61] "\n                                Bobby Hendricks\n                            "                            
##  [62] "\n                                Ed Townsend\n                            "                                
##  [63] "\n                                Jerry Lee Lewis And His Pumping Piano\n                            "      
##  [64] "\n                                Toni Arden\n                            "                                 
##  [65] "\n                                The Platters\n                            "                               
##  [66] "\n                                Jim Backus & Friend\n                            "                        
##  [67] "\n                                The Four Preps\n                            "                             
##  [68] "\n                                Dean Martin\n                            "                                
##  [69] "\n                                The Honeycones\n                            "                             
##  [70] "\n                                The Playmates\n                            "                              
##  [71] "\n                                Frank Gallup\n                            "                               
##  [72] "\n                                Connie Francis\n                            "                             
##  [73] "\n                                Oscar McLollie and Jeanette Baker\n                            "          
##  [74] "\n                                Eydie Gorme\n                            "                                
##  [75] "\n                                Sam Cooke\n                            "                                  
##  [76] "\n                                Paul Anka\n                            "                                  
##  [77] "\n                                The Diamonds\n                            "                               
##  [78] "\n                                Bobby Freeman\n                            "                              
##  [79] "\n                                David Seville\n                            "                              
##  [80] "\n                                Chuck Berry\n                            "                                
##  [81] "\n                                Chuck Berry\n                            "                                
##  [82] "\n                                Bill Doggett\n                            "                               
##  [83] "\n                                Lee Andrews And The Hearts\n                            "                 
##  [84] "\n                                Eddie Cochran\n                            "                              
##  [85] "\n                                The Daddy-O's\n                            "                              
##  [86] "\n                                Jackie Wilson\n                            "                              
##  [87] "\n                                Kitty Wells\n                            "                                
##  [88] "\n                                The Upbeats\n                            "                                
##  [89] "\n                                Jim Reeves\n                            "                                 
##  [90] "\n                                The Ames Brothers\n                            "                          
##  [91] "\n                                Joe South\n                            "                                  
##  [92] "\n                                The Everly Brothers\n                            "                        
##  [93] "\n                                Jimmie Rodgers\n                            "                             
##  [94] "\n                                Marty Robbins\n                            "                              
##  [95] "\n                                Fats Domino\n                            "                                
##  [96] "\n                                Thurston Harris\n                            "                            
##  [97] "\n                                Robert & Johnny\n                            "                            
##  [98] "\n                                The Ames Brothers\n                            "                          
##  [99] "\n                                Billy Williams\n                            "                             
## [100] "\n                                Frankie Vaughan\n                            "
myartists <- sub("^\\s+", "", myartists)
myartists <- sub("\\s+$", "", myartists)
myartists
##   [1] "Ricky Nelson"                               
##   [2] "Perez Prado And His Orchestra"              
##   [3] "Bobby Darin"                                
##   [4] "Elvis Presley With The Jordanaires"         
##   [5] "Kalin Twins"                                
##   [6] "Duane Eddy His Twangy Guitar And The Rebels"
##   [7] "The Coasters"                               
##   [8] "Jack Scott"                                 
##   [9] "The Johnny Otis Show"                       
##  [10] "Peggy Lee"                                  
##  [11] "Frankie Avalon"                             
##  [12] "Jimmy Clanton And His Rockets"              
##  [13] "Patti Page"                                 
##  [14] "Pat Boone"                                  
##  [15] "Jerry Butler and The Impressions"           
##  [16] "The Danleers"                               
##  [17] "Jody Reynolds"                              
##  [18] "The Elegants"                               
##  [19] "Doris Day"                                  
##  [20] "Bobby Freeman"                              
##  [21] "Johnny Cash And The Tennessee Two"          
##  [22] "Johnny Mathis"                              
##  [23] "The Olympics"                               
##  [24] "Sheb Wooley"                                
##  [25] "Chuck Willis"                               
##  [26] "Poni-Tails"                                 
##  [27] "The Crickets"                               
##  [28] "Jimmie Rodgers"                             
##  [29] "The Four Lads"                              
##  [30] "Dean Martin"                                
##  [31] "Big Bopper"                                 
##  [32] "Don Gibson"                                 
##  [33] "Tony And Joe"                               
##  [34] "Elvis Presley With The Jordanaires"         
##  [35] "Bobby Day"                                  
##  [36] "Gerry Granahan"                             
##  [37] "Perry Como"                                 
##  [38] "Buddy Knox with the Rhythm Orchids"         
##  [39] "Pat Boone"                                  
##  [40] "Bobby Hamilton"                             
##  [41] "Buddy Holly"                                
##  [42] "The Slades"                                 
##  [43] "Clyde McPhatter"                            
##  [44] "Jan & Arnie"                                
##  [45] "The Diamonds"                               
##  [46] "Gino & Gina"                                
##  [47] "Dion & The Belmonts"                        
##  [48] "Dean Martin"                                
##  [49] "The Everly Brothers"                        
##  [50] "Jimmy Bowen with the Rhythm Orchids"        
##  [51] "The Kirby Stone Four"                       
##  [52] "The Rinky-Dinks"                            
##  [53] "Nat King Cole"                              
##  [54] "Domenico Modugno"                           
##  [55] "The Three G's"                              
##  [56] "Jack Scott"                                 
##  [57] "Eydie Gorme"                                
##  [58] "The Crickets"                               
##  [59] "Tony Bennett"                               
##  [60] "Bobby Day"                                  
##  [61] "Bobby Hendricks"                            
##  [62] "Ed Townsend"                                
##  [63] "Jerry Lee Lewis And His Pumping Piano"      
##  [64] "Toni Arden"                                 
##  [65] "The Platters"                               
##  [66] "Jim Backus & Friend"                        
##  [67] "The Four Preps"                             
##  [68] "Dean Martin"                                
##  [69] "The Honeycones"                             
##  [70] "The Playmates"                              
##  [71] "Frank Gallup"                               
##  [72] "Connie Francis"                             
##  [73] "Oscar McLollie and Jeanette Baker"          
##  [74] "Eydie Gorme"                                
##  [75] "Sam Cooke"                                  
##  [76] "Paul Anka"                                  
##  [77] "The Diamonds"                               
##  [78] "Bobby Freeman"                              
##  [79] "David Seville"                              
##  [80] "Chuck Berry"                                
##  [81] "Chuck Berry"                                
##  [82] "Bill Doggett"                               
##  [83] "Lee Andrews And The Hearts"                 
##  [84] "Eddie Cochran"                              
##  [85] "The Daddy-O's"                              
##  [86] "Jackie Wilson"                              
##  [87] "Kitty Wells"                                
##  [88] "The Upbeats"                                
##  [89] "Jim Reeves"                                 
##  [90] "The Ames Brothers"                          
##  [91] "Joe South"                                  
##  [92] "The Everly Brothers"                        
##  [93] "Jimmie Rodgers"                             
##  [94] "Marty Robbins"                              
##  [95] "Fats Domino"                                
##  [96] "Thurston Harris"                            
##  [97] "Robert & Johnny"                            
##  [98] "The Ames Brothers"                          
##  [99] "Billy Williams"                             
## [100] "Frankie Vaughan"

Did it work?

length(mysongs)
## [1] 100
length(myartists)
## [1] 100

Question 1.

1a. Write a function that takes one date as input, and it extracts the song titles for that week.

1b. Use the sapply and the unlist function to get a vector of all of the song titles for all of the weeks.

1c. Write a function that takes one date as input, and it extracts the artists for that week.

1d. Use the sapply and the unlist function to get a vector of all of the artists for all of the weeks.

Hint: It might be helpful to use:

mydates <- seq(as.Date("1958-08-09"), as.Date("2016-10-08"), by = "week")

Question 2.

Take your data from Question 1 and build a data.frame with four columns: the artists, the songs, the weeks, and the rank within the week. You will need to build another function to extract the positions.

Questions 3-10.

Ask 8 questions about the Billboard data, and answer each question, using the data.frame that you have built. It would be nice to make some visualizations about the data, for some of your questions. Have fun!