Premise

A teenage girl runs away from home to save her dog from being put down but gets caught in a tornado and is whisked away to a far off magical land.

Theatre

There’s an electric atmosphere in the London Palladium that I haven’t experience at any other London theatre. Touted as the most famous theatre in the world due to it’s long history of hosting the biggest names in showbiz, and therefore some of the biggest shows including the Royal Variety Performance, I did wonder if I would have felt any different about the place had I not previously known of its legacy. I probably still would’ve been awe struck as it is, to me, one of the grandest of any of the theatres I’ve been to.

I was sat in the Stalls, Row D, seat 19. At 6′ tall legroom was bearable with about an inch between my knees and the seat in front though I was fortunate to be able to tuck my legs under said seat for confort (otherwise I think I would’ve ended up sporting a cramp). Being so close to the stage meant that when the stage was flat I was unable to see the feet of the actors when they went towards the back of the stage. The seats in row D are not at all staggered with those in row C,  meaning the head of the person directly in front of me was always in the way (other than when they slumped into their seat). The rake is also almost non-existent for the first four rows at least. I ended up with a slight ache in my neck from looking upwards but nothing major (might be different for those shorter than I).

Plot

I haven’t watched the movie in its entirety since I was a kid but what I remember is it  that it started in black and white and then switched over to technicolour when Dorothy, her dog Toto, and their house land in Oz after some freak tornado steels them away, and killing the wicked witch of the east in the process. Then something about munch-kins,  the evil witch’s even wickeder witchy sister, a good witch, a yellow brick road, a scarecrow, a tin man, a lion, a wizard and some ruby slippers. Oh, and some songs.

All of the elements above are in the theatrical production. The plot basically revolves around getting Dorothy back home to Kansas by realising that home is where the heart is and that she has it within her to rescue herself.

Verdict

I was surprised at how much I loved the opening scenes pre-OZ. The opening song “Nobody understands me” is great (I love that type of music, I’ll put a link to the song after this paragraph), the stage scenery (windmill, house, carts etc.) drew me in at once and were perfect with projected video adding even more details to this immersive experience.

Then we get to Oz and it all gets super-cheese. Now, I like overly sentimental dialogue to a rather embarrassingly large degree but for some reason this just irritated me. I think I wanted more depth which in retrospect is more of a failing on my part because this musical is meant to be a family show and rather light entertainment and so in that context I think it succeeds. I really enjoyed the Scarecrow, the Tin-man, and most of all the Lion who had me laughing throughout. I can  identify with each of the characteristics that those three feel they are missing (intelligence, emotions and bravery) and therefore I liked them as characters (I know, it’s not objective but I always like characters I can identify with in some way).

Having seen Wicked a few years previous and having only midly enjoyed it, I must confess that in comparison to the The Wizard of OZ, Wicked is the superior musical in my opinion with a vastly superior staging of the machanical Wizard of Oz than is achieved at the Palladium. Again however, I think they each have a different audiance in mind, and so with that context it’s not fair to directly compare them other than to say that I personally preferred Wicked for it’s attempt at not having so many 2D characters.

Overall, it was a fun evening with some surprises with actors being suspending from the air (I won’t spoil those with more details because I didn’t know about them before hand and they proved to be the highlights for me). I liked the ambiguity of Oz being either fantasy or reality. I liked Dorathy effectively realising that she has the resources to save herself and I think that aspect of the musical raises it up several levels in my estimation. Plus there’s about 3 or 4 songs which I adore and will be looking to download them in the near future. I wouldn’t go out of my way to watch it again but I didn’t hate it and that’s something

Premise

A case of mistaken identity about two sets of twins who are are separated at birth.

Theatre

This was my first time into one of the auditoriums of the National Theatre. I initially got a bit lost inside the building as I couldn’t see which was the way to the Olivier auditorium but thankfully a staff member saw the confusion in my face and helped me out. In the main foyer there was a band playing some songs (country music) and it made me wish that I had arrived a bit earlier so that I could’ve heard more of their set as it was rather pleasant.

I was sat in the Stalls, Row L, Seat 6. At 6 foot tall the legroom was good, the stagnation of seats between rows very good, and the rake excellent. I had a fantastic and unobstructed view of the entire stage. I really like the layout of this theatre and got the impression that one would have a decent view from almost any seat given that the stage is rounded and so those on the sides (like my seat was) won’t miss anything. I wish more theatre’s in the West End were like this one.

As always, for a wider variety of seating opinions, check out the Theatre Monkey website.

Plot

The play starts off with an elderly Syracusian man being arrested for his presence in Ephesus (where no Syracusian merchants are allowed by law). In order to escape execution he must pay a fine of 1000 marks.

He tells the Duke his sad story of when he was young he had purchased the twins of a poor woman to be slaves to his own twin sons. Soon afterwards, he, his wife, his baby twin sons and the baby twin slaves make a sea voyage during which there is a storm. He is rescued with one of his twin sons and one of the twin slaves. His wife is also rescued but by another boat with the other twin son and twin slave. However, he never sees her, or the children she took with her, again. The boys he saved grow into men and set off to find their brothers but he had not heard from them in years and so set off to find them, which is how he ended up on Ephesus.

The rest of the plot then revolves around the Syracusian twins being mistaken for the Ephesian twins, and vice-versa, throughout the rest of the play. This results in wrongful beatings, a near-seduction (at least in one case as I think the other might’ve been sucesful), an arrest, accusations of infidelity, theft, madness, and demonic possession.

Verdict

When the play started I found it very difficult to understand what anyone was saying because they were using old Shakespearian English. I never liked studying Shakespeare when I was at school because I never understood any of it (though to be fair I didn’t understand most of what I read in my early teens, and it wasn’t until I was 18 that I started to read books independently).

This was the first time I’ve  ever seen any Shakespeare live and initially thought it would be a dull evening and a waste of time given my inability to understand what was being said. However, once I understood the rhythm of the speech I was able to get the gist of what was spoken. Anything I didn’t understand from the speech I was able to infer from the actions and reactions of the cast and audience.

There is something very entertaining and endearing about hearing long multisyllabic words spoken in a non-posh accent. I wish I knew how to construct and speak sentences full of such lovely words.

The production uses contemporary scenery and props to escape the trappings of being set during a much ear. As I understand it, the original  play was also contemporary and so therefore I have no issues with the update (plus it makes it easier to understand what is going on in my opinion). However, even I’m mistaken about the original play being contemporary for it’s time, as a novice, I still found the update helpful.

Most of the Play was predictable but still enjoyable. I even liked the happy ending even though I usually prefer bitter sweet endings. I thought the leads did a fantastic job and really made be believe in the characters they were portraying. Also, the scene changes had a set of musicians appear on stage sing a short contemporary and popular song (in a foreign language) while the set is changed behind them.

Overall, I did enjoy the play and, if given the opportunity, would now happily watch another Shakespearian play as I’m beginning to see what all the fuss about him is.

Premise

A musical about a young girl who wishes to become a firework-maker but is instead told by her father that her destiny is to find a husband and have children. *sad face*

Theatre

The Bloomsbury theatre is owned by UCL (University Collage London) and located on Gordon Street in Central London’s Bloomsbury area, near Euston, Euston Square, and Warren Street tube stations, behind Tottenham Court Road. Being a modern theatre (about 43 years old) it has a seating layout for 535 seats which provide excellent legroom and a fantastic rake to provide a clear view of the stage. We were sat in the stalls, seats I30 and I31 and would happily sit there again.

Audience

As one might expect from a production aimed at children, there were many of them about. Throughout the show we could hear them talking quietly to one another, rustling bags of crisps, rustling bags of sweets, and making frequent trips to the toilet. However this was not unexpected and so one just tries to ignore it. Far more difficult to ignore and excuse are the adults who whisper far too loudly to one another or rustle their own bags of food with far too little thought for those around them.

Whenever I think of badly behaved audiences I have a automatic impulse to have a quick read of the What’s On Stage discussion board to reassure myself that I’m not the only one who gets annoyed by this behaviour: https://www.whatsonstage.com/board/index.php?/topic/22065-bad-behaviour-at-a-show

Plot

A young girl called Lila wishes to become a firework-maker just like her father. However her father believes this to be an unsuitable job for a girl and refuses to make her his apprentice and tell her the secret all firework-maker’s must know. Lila is quite capable and has produce innovative fireworks of her own but longs to know what the secret is. Her friend Chulak on hearing of Lila’s sadness tricks Lila’s father into telling him the secret which he then tells Lila: she must journey to acquire Royal Sulphur from Razvani the Fire-Fiend at Mount Merapi. Lila sets off on her adventure and encounters four pirates on her way who at first want to kill her but change their minds when she saves their lives.

Chulak and the elephant he’s taking care of, Hamlet, go after Lila to help her because there was some information Lila’s father had not divulged to Chulak and which could lead to Lila’s death. Specifically she needs three gifts for the fire-fiend and some magic water from the Goddess of the Emerald Lake to protect her. Chulak is able to acquire the water because his reason’s are not selfish and arrives just in time to give it to Lila. However, Lila doesn’t know what the three gifts she needs are but the fire-fiend recognises the three gifts in her but still doesn’t give her the Roayl Sulphur and disapears.

When Lila, Chulak and Hamlet return home they find that Lila’s father has been imprisoned for the disappearance of Hamlet. The King agrees to release her father if she can win his firework contest which will have the greatest firework-maker’s in the world in attendance. Through hard work Lila is able to win the contest by telling her story of wanting to become a firework-maker through her firework display. With her father released he tells her that he was wrong to deny her ambition and that the three gifts she didn’t know about but had in great supply where internal ones: they are talent, courage, and luck. She has talent, having worked with her father at firework-making for many years; courage, for having undertaken the journey; and good fortune, which lies in having loyal friends, Chulak and Hamlet.

Verdict

Based on the Philip Pulman story with the same name, I had high hopes, especially as Pullman was quoted to have said “One of the best productions of my work I’ve ever seen!”. I enjoyed having the musicians up on stage and being part of the natural background or having the actor’s play instruments, it just adds that extra something to the show which makes it feel a bit special. Furthermore the use of puppets to show Lila going up and falling down the mountain was just so inspiring with excellent work by everyone involved.

In terms of the music, the opening song (heard in the trailer below) was rather good and played several times throughout the show. I can’t really remember what the other songs were but I’m not sure if that’s because they weren’t particularly noteworthy or because in comparison to the opening number they just didn’t have as much impact.

Overall I though the show was OK. I wouldn’t go out of my way to watch it again unless I needed something to take children to but that’s just because I prefer more multi-level plots.  The show itself is well written and well produced

Premise

A Smallville-esque retelling of the Arthurian legend set in a time of myth and magic when Arthur is still a Prince and Merlin his faithful servant.

Plot

The series story arc this year revolved around advancing to the next stage of the legend with Morgana fully embracing her evil side to plot the murder of Arthur, Guinevere becoming Arthur’s betrothed and her betrayal with Lancelot, Arthur taking on the political responsibilities of being the new King, and Merlin beginning to truly earn a measure of respect and leading Arthur to the Sword in the Stone.

Verdict

I thought that this was the best series to date. Any episode where Merlin becomes the aged Emris was an absolute delight with the show achieving some of its best comedy right there. Morgana is a difficult character to have sympathy with but Angel Coulby plays the role so well with glimpses of vulnerability that one can not help but stay interested in her story. For the Knights of Camelot, Eoin Macken’s Sir Gwaine brings a charming roguish qulity which makes him the only one you’d probably actually want to hang out with.  Bradley James does an excellent job as King Arthur Pendragon throughout especially in his more quiet and depressed states where he really sells those emotions to the audience.

Some of the low points for me were whenever Arthur is either dismissive or rude towards Merlin. At this point of being in the fourth series one would hope that there would’ve been enough character development for him to be at least more outwardly respectful towards Merlin instead still being just quietly respectful of him on the inside. I do recognise that his attitude has improved with each successive series but this attatude is really rather grating at this point with the comedic effect delivering diminishing returns. Also, even though Guinevere didn’t truly betray Arthur of her own free will, no one knows that, and so him still wanting her to be his wife just doesn’t ring true to me but if he loves her enough to forgive her then maybe that’s all that truly matters.

I would like to see more magical fights in the future but I’m guessing that the budget won’t allow for that. What would be really interesting would be for Arthur to find out about Merlin’s magical powers and see how that affects their relationship but that would probably be saved for either the final or penultimate series of the show. Overall, a great series and I’m now really pumped for series five to air sometime in 2012!

Premise

Based on the Dicken’s novel of the same name, a young man apprenticing as a blacksmith is bestowed a large deal of money from an unknown benefactor. He goes to London to become a gentleman but is haunted by his past.

Plot

Young Pip is an orphan who lives with his elder sister and her husband, Joe, who is a blacksmith. One day Pip encounters an escaped convict and is scared into stealing a file for him so he can break out of his chains. Pip not only steels the file but also steels a piece of pie which he gives to the convict because he thought he might be hungry. Eventually the convict is caught though he does not reveal Pips involvement.

Later, Pip is commanded by his sister and uncle to visit Miss Havisham, an old and wealthy spinster, who has asked for a young boy from the village to be a playmate for her adopted daughter, Estella. Pip has his mind opened up to what the world is capable of offering him but at the same time is treated poorly by Estella who belittles him and even makes him cry by slapping him hard after he calls her lonely (and after that he promises he will never cry for her again). However there is also some tenderness such as when a rich boy tries to bully Pip but gets a good punch in return. Estella gives Pip a kiss for this action, which is seen by Miss Havisham.

Miss Havisham invites Pip and Joe to her house and offers to pay for Pip’s apprenticeship as a Blacksmith. Pip is crestfallen as he had hoped for something grander after she had opened his eyes to what the world could offer.

Years pass and Pip has grown and is doing well as the apprentice of a blacksmith. He has to put up with the bully Bulge who also works there but Pip just keeps a level head about him and gets his work done. However, a lawyer turns up and informs Pip that he is to receive a large sum of money from an anonymous benefactor and is to go to London to become a gentleman.

The rest of the plot revolves around Pip mistakenly believing Miss Havisham to be his benefactor and furthermore believing that she intends him and Estella to be together, Pip becoming a gentleman thanks to a new and true friend (the boy who Pip punched as a child), turning his back on Joe, having his heart broken when Estella is to be married to a man he can not stand, falling into debt, and finally learning that the convict he stole for so many years past is infact his true benefactor.

Verdict

I love period drama – everything just feels so magical and other-worldly without being too far removed. Even having read the book and watched several other versions of this story I still enjoyed it. The real moving part for me was when the convict takes the file he had scared Pip into stealing and is then in additional is offered a piece of pie out of the goodness of Pip’s heart. Everything about that was perfect to me – volumes were spoken in the silence of that scene. Overall, this production was a success in my book with my favourite interpretations being performed by the actors playing the roles of young Pip, the convict and, Miss Havisham.

Premise

The English National Ballet presents two hours showcasing some of Gershwin’s most famous and iconic musical compositions being performed with a live orchestra and a variety of choreographed interpretive dances.

Gershwin

An American composer and pianist, the works of George Gershwin are widely known and respected. Among his most popular tunes are Rhapsody in Blue (1924) and An American in Paris (1928). He wrote music for a number of Broadway musicals (such as Funny Girl and Crazy For You), Operas (Blue Monday, and Porgy and Bess),  films (e.g. Shall We Dance ) and stand-alone pieces (e.g. Preludes For Piano).

Theatre

This was my second time inside the London coliseum and I still think it’s among the most beautiful of auditoriums I’ve been in. I was sat in the stalls, row G, seat 8. Even though the seat is on the side it still affords a fantastic view of the stage with only the very extreme right being out of sight (which for this production was a non-issue). At 5′ 11” legroom was decent enough. The rake is good such that the people in front didn’t obstruct my view. The seats between rows are slightly staggered which also aids an obstructed view.

Audience

I was unfortunate enough to experience some annoying behaviour from a small minority of the audience members sat around me. A couple in the row in front of me and to the right whispered to one another throughout the show and blocked the view of the people directly behind them whenever they kissed or one rested their head on the shoulder of the other. The father of the family sat next to me would make sarcastic and loud comments about the performance to his wife sat a couple seats away from him which was very distracting. Finally there was a couple of elderly ladies sat behind me at the end of the row who also made comments (mostly complimentary) to one another about the performance but didn’t even attempt to whisper and instead spoke at a normal conversational volume.

I stayed quiet for as long as I possible could, doing my best to ignore this rude and inconsiderate behaviour, and just when I thought I could take no more, I said absolutely nothing. I just quietly sat there and did my best to enjoy the performance which is what I think the rest of the well behaved audience around me opted for (though a few brave souls did shush the offending parties on several occasions and others gave them the evil eye).

An entertaining thread documenting some of the experiences theatregoers have had with badly behaved audiences can be found in the following thread from whatsonstage.com’s discussion forum: https://www.whatsonstage.com/board/index.php?/topic/2984-badly-behaved-audiences/

Verdict

Having obtained a couple of cheap tickets I thought it would be an interesting experience to attend this type of show where an orchestra plays some popular compositions of a well known composer whilst a company of dancers interpret it live on stage. The closest thing I have to compare this with would be the multitude of dance shows on TV such as “So You Think You Can Dance” and “Strictly Come Dancing”. I’m a bit of a fan of SYTYCD and love how it incorporates a wide variety of different dancing and musical genres to keep the entertainment factor high. This is effectively what Strictly Gershwin does and I must say is very successful at accomplishing.

I must confess that I am not very knowledgeable about the art forms I witnessed at the London Coliseum but will say that I was both highly impressed with the dancers and blown away by orchestra. Words seem to fail me as I don’t have the vocabulary to do justice to the performance.

The dancers were flawless to my untrained eyes. They performed a variety of choreographed set pieces from several appropriate genres (ballet, ballroom and tap dance were the three easiest styles for me to pick out). The way the ballerina’s almost floated across the stage by standing on the tips of their toes and then in quick succession alternating each foot hitting the ground and the other in the air was seriously impressive (and slightly amusing).

The conductor of the orchestra had a bit of fun too doing his own little funny dances to amuse the audience between sets. The orchestra itself was, to my untrained ears, flawless and the singers pitch perfect.

Overall, I had a great time at Strictly Gershwin, much better than I had anticipated and will certainly be less hesitant to attend these types of performances in the future!

Footnote

I used this post http://wpbtips.wordpress.com/2011/06/29/inserting-grooveshark-audio-players/ to figure out how to embed the grooveshark flash content above into a wordpress.com blog post.

Premise

A musical, based on the book of the same name, set in the early 1900′s about a group of children who decide to play make-believe about pirates, amazons and treasure over the period of several summer days and evenings.

Theatre

The Vaudeville is a nice cosy theatre with three levels. The staff are pleasant and it’s located very close to Covent Garden, Embankment and Charring Cross stations. We were sat in the Dress Circle (second level) in seats D16  and D15 (which have good enough leg room for someone about 6 foot tall). Very luckily we had short children sat in front of us which gave us a full view of the stage. During this production the cast members sometimes left the stage and went among the audience and part of that was missed from these seats. If I were to go again I’d want to be sat in the stalls (where I have sat in the past for this theatre and must say give an excellent view). As always, check out the fantastic Theatremonkey website for a wider variety London theatre seating opinions.

Audience

Being a show aimed at children there were lots of families in attendance but also a good deal of adults without children. Based on the audience reactions and a quick look on twitter it would seem everyone was having a very good time. Whilst people often whisper to each other during a show (which is considered very rude by most decent theatre-goers), one thing that short children will do is actually stand up for a short burst if there is something they can’t see very well during an entertaining part of the show. Rather more annoyingly however is when their parents don’t immediately correct this behaviour as it spoils the view for everyone immediately behind them, all the way to the back of the auditorium.

Plot

Four young siblings set sail for an island in the hope of finding adventure. They call themselves the Swallows and consist of “captain” John who is the eldest brother and the one in charge, “first mate” Susan who tries to make sure everyone is OK, “able seaman” Titty who thirsts for adventure and  youngest brother Roger who is the “ship’s boy” and almost eight years old. As they land on the island they meet a couple of “Amazons” who are slightly older and consist of “captain” Nancy and her sister Peggy.

Both the Swallows and Amazons are initially at odds with one another until they realise they have a common enemy in that of Captain Flint (the uncle of the Amazon’s who has been neglecting his niece’s in order to finish a book he has been writing, and who blames the Swallows for a firework going off on his houseboat, though that was actually the Amazons). In order to decide who will lead the the attack on Captain Flint they decide to have a competition that whoever can capture the other’s boat first will have the honour of being the flagship.

Whilst the Swallow’s make preparations they are asked to make Captain Flint aware that someone may try to burgle his houseboat. Captain Jonn sets sail alone to inform Captain Flint of this information but is unable to do so because Captain Flint won’t hear him out (still believing The Swallows are responsible for the Firework going off on his houseboat) and calls John a liar (which John is visibly upset about).

The rest of the plot revolves around the Swallows capturing the Amazon’s boat, the Amazon’s graciously conceding defeat, the Swallows being wrongfully accused of breaking into Captain Flint’s houseboat, Captain Flint apologising to John for calling him a liar and Titty being a hero to help locate the stolen property. Finally Captain Flint makes proper amends by agreeing to play with both the Swallow’s and the Amazons.

Here’s a trailer (which does not do the show justice at all but it’s the best I could find!)

Verdict

I went into this with very low expectations but these were vastly exceeded by the pure brilliance and fun factor of the musical. It started off a bit slow but soon kicked up a gear when the Swallow’s left home for their camping adventure. From this point on it was fantastic.

The music is catchy with my favourite being when the theme songs of both the Swallows and the Amazons are song over one another (I wish I knew where to find it). I love it when there are musician’s on stage who blend into the background and that occurs here. There is very clever uses and realisations of the simple props to hand, such as when a telescope is used one of the stage hands produces a large hollow circle (with simultaneous motion to that of the telescope holder) to show what is being magnified, and the parrot whose eyes and beak are made up of pliers. To simulate the sound of fire on stage, several stage click their fingers in an arrhythmic pattern and is actually very convincing. The actor’s do a decent job with the woman playing the role of Susan deserving special praise not only for her singing but also for giving the most convincing performance of the character she was portraying.

Overall a most excellent production and one I would love to see again (if it weren’t closing this week!).

Introduction

I was looking at a list of the top grossing films of all time (available from boxofficemojo.com) and was wondering what kind of graphs I would come up with if I had that kind of data. I still don’t know what kind of graphs I’d construct other than a simple barplot but figured I’d at least get the basics done and then if I feel motivated enough I could revisit this in the future.

Objective

Scrape the information available on http://boxofficemojo.com/alltime/world into R and make a simple barplot.

Solution

This is probably one of the easier scraping challenges. The function readHTMLTable() from the XML package does all the hard work. We just point the url of the page we’re interested in and feed it into the function. The function then pulls out all tables on the webpage as a list of data.frames. We then choose which data.frame we want. Here’s a single wrapper function:

box_office_mojo_top <- function(num.pages) {
  # load required packages
  require(XML)

  # local helper functions
  get_table <- function(u) {
    table <- readHTMLTable(u)[[3]]
    names(table) <- c("Rank", "Title", "Studio", "Worldwide.Gross", "Domestic.Gross", "Domestic.pct", "Overseas.Gross", "Overseas.pct", "Year")
    df <- as.data.frame(lapply(table[-1, ], as.character), stringsAsFactors=FALSE)
    df <- as.data.frame(df, stringsAsFactors=FALSE)
    return(df)
  }
  clean_df <- function(df) {
    clean <- function(col) {
      col <- gsub("$", "", col, fixed = TRUE)
      col <- gsub("%", "", col, fixed = TRUE)
      col <- gsub(",", "", col, fixed = TRUE)
      col <- gsub("^", "", col, fixed = TRUE)
      return(col)
    }

    df <- sapply(df, clean)
    df <- as.data.frame(df, stringsAsFactors=FALSE)
    return(df)
  }

  # Main
  # Step 1: construct URLs
  urls <- paste("http://boxofficemojo.com/alltime/world/?pagenum=", 1:num.pages, "&p=.htm", sep = "")

  # Step 2: scrape website
  df <- do.call("rbind", lapply(urls, get_table))

  # Step 3: clean dataframe
  df <- clean_df(df)

  # Step 4: set column types
  s <- c(1, 4:9)
  df[, s] <- sapply(df[, s], as.numeric)
  df$Studio <- as.factor(df$Studio)

  # step 5: return dataframe
  return(df)
}

Which we use as follows:

num.pages <- 5
df <- box_office_mojo_top(num.pages)

head(df)
# Rank Title Studio Worldwide.Gross Domestic.Gross Domestic.pct Overseas.Gross Overseas.pct Year
# 1 1 Avatar Fox 2782.3 760.5 27.3 2021.8 72.7 2009
# 2 2 Titanic Par. 1843.2 600.8 32.6 1242.4 67.4 1997
# 3 3 Harry Potter and the Deathly Hallows Part 2 WB 1328.1 381.0 28.7 947.1 71.3 2011
# 4 4 Transformers: Dark of the Moon P/DW 1123.7 352.4 31.4 771.4 68.6 2011
# 5 5 The Lord of the Rings: The Return of the King NL 1119.9 377.8 33.7 742.1 66.3 2003
# 6 6 Pirates of the Caribbean: Dead Man's Chest BV 1066.2 423.3 39.7 642.9 60.3 2006

str(df)
# 'data.frame': 475 obs. of 9 variables:
# $ Rank : num 1 2 3 4 5 6 7 8 9 10 ...
# $ Title : chr "Avatar" "Titanic" "Harry Potter and the Deathly Hallows Part 2" "Transformers: Dark of the Moon" ...
# $ Studio : Factor w/ 35 levels "Art.","BV","Col.",..: 7 20 33 19 16 2 2 2 2 33 ...
# $ Worldwide.Gross: num 2782 1843 1328 1124 1120 ...
# $ Domestic.Gross : num 760 601 381 352 378 ...
# $ Domestic.pct : num 27.3 32.6 28.7 31.4 33.7 39.7 39 23.1 32.6 53.2 ...
# $ Overseas.Gross : num 2022 1242 947 771 742 ...
# $ Overseas.pct : num 72.7 67.4 71.3 68.6 66.3 60.3 61 76.9 67.4 46.8 ...
# $ Year : num 2009 1997 2011 2011 2003 ...

We can even do a simple barplot of the top 50 films by worldwide gross (in millions) :

 require(ggplot2)
 df2 <- subset(df, Rank<=50)
 ggplot(df2, aes(reorder(Title, Worldwide.Gross), Worldwide.Gross)) +
   geom_bar() +
   opts(axis.text.x=theme_text(angle=0)) +
   opts(axis.text.y=theme_text(angle=0)) +
   coord_flip() +
   ylab("Worldwise Gross (USD $ millions)") +
   xlab("Title") +
   opts(title = "TOP 50 FILMS BY WORLDWIDE GROSS")

Here’s an excerpt:

A San Francisco cable car holds 60 people. This blog was viewed about 3,000 times in 2011. If it were a cable car, it would take about 50 trips to carry that many people.

Click here to see the complete report.

Introduction

R-bloggers.com is a blog aggregator maintained by Tal Galili. It is a great website for both learning about R and keeping up-to-date with the latest developments (because someone will probably, and very kindly, post about the status of some R related feature). There is also an R-bloggers facebook page where a number of articles from R-bloggers are linked into its feed. These can then be liked, commented upon and shared by other facebook users. I was curious if anyone had commented on any of my R posts which had been linked into this facebook feed but it is a very tedious process to have to manually and continually click the ‘load more’ button to load more posts into the facebook wall page and scan for one of my posts.

Objective

Automatically scrape the content off of the R-bloggers facebook wall page via XPath and structure it into a dataframe in order to see if anyone has made any comments on one of my posts, or liked it or shared it.

Initial Thoughts

I have posted previously about using the Facebook Explorer API to get data from facebook. However there is a issue whereby a set of random posts may not be returned by the API. Given that I’m specifically interested in a small subset of posts, this issue makes it unsuitable for me to use the API as there is a chance I might miss something interesting. (My feeling is this has something to do with privacy issues but I’m not sure because then surely I wouldn’t be able to see a private post at all whether it’s through the facebook wall or Graph API, unless the API is more strict about privacy).

I could try logging directly into Facebook using RCurl and doing things like setting cookies but that would require me having to first learn HOW to set cookies in RCurl (and feeling motivated enough to spend the extra time required to do it). Seeing as I really want to spend the majority of my spare programming time learning python, I’m going to give this one a miss for now.

Therefore I want to do this scraping of data using the skills I already have (which is a rather basic understanding of XPath via the XML package). I was tempted to learn about setting cookies with RCurl but it’s Friday and that means I just want the weekend to start already…

Limitations

Links to blog posts on the Facebook wall often do not give information about the original author of the blog. This is rather annoying because it means that some web-crawling is necessary to find out who wrote the post instead of that information being readily available in the first instance. I’m going to limit my code to only crawling for extra information from R-bloggers.com links because it is very easy to scrape data off that website via XPath (and saves me writing lots of code to try and work with other types of websites).

The R-bloggers facebook page has wall posts going back to January 2010. Prior to September 2011 blog posts pointed to the “notes” page on facebook. This prevents me getting extra data about the blog post because I can’t automatically navigate to those facebook pages. From Septermeber 2011 onwards however the blog posts point to R-bloggers.com and so these can be scraped for further information rather easily. Luckily I only started posting in November 2011 so this isn’t an issue for me.

Not all wall posts indicate how many comments they have if there are only a few comments. Not sure how to get round this, might have to write “at least 1 comment” for this situation maybe.

Most of the wall posts are made up of links to R-bloggers.com and various messages by Facebook users. Instead of filtering out, I’m just going to grab AS MUCH INFORMATION off of the wall feed as I can and then filter at the end. I’ll put the unfiltered information into a csv file for anyone that may want it and post it up on github.

Method

The easiest method would be to log into Facebook via the browser, navigate to the R-bloggers Facebook page, use the socialfixer.com browser add-on “Better Facebook” to automatically and painlessly load all posts in the R-bloggers feed going back to January 2010 and then save that page to the hard drive using, in google chrome browser terminology, the “Web Page, Complete” option (NOT the “Web Page, HTML Only” option because for some reason that won’t work well with my code).

Once the data is in a html file, use XPath expressions via Duncan Temple Lang’s XML package to extract whatever information I can in the first instance and store into a data.frame.

Once this initial data is in place, I will crawl any posts which link to R-bloggers.com and extract extra information about the post (e.g. Author, original publication date, post title, etc.). I will merge this data with the already constructed data.frame above.

I will then save this data.frame to a .csv file in case anyone else wishes to analyse it (thus saving them some time). Finally I will subset the data.frame to only posts that link to one of my blog posts and inspect the output.

Solution

source_https <- function(url, ...)  {
  # load package
  require(RCurl)

  source_script <- function(u) {
    # read script lines from website using a security certificate
    script <- getURL(u, followlocation = TRUE, cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl"))

    # parse lines and evaluate in the global environement
    eval(parse(text = script), envir= .GlobalEnv)
  }

  # source each script
  sapply(c(url, ...), source_script)
}

Following the procedure describe in the Method section above:

1. Log into facebook
2. Naviagate to the R-bloggers facebook wall
3. Load data as far back as you like. I used the Better Facebook browser add-on tool to automatically load data right back to January 2010.
4. Save this webpage as a “complete” html file.
5. Run the following code, selecting the location of the html file when prompted:

source_https("https://raw.github.com/tonybreyal/Blog-Reference-Functions/master/R/RBloggersFBXScraper/rbloggersFBXScraper.R")
df <- rbloggersFBXScraper()

Depending on your internet connection this could take quite some time to complete because it has to crawl the R-bloggers website for extra information about links posted since September 2011. To save you some time I’ve saved ALL the data which I have scraped into a single csv file. Here’s how to use it:

library(RCurl)
csv.location <- "https://raw.github.com/tonybreyal/Blog-Reference-Functions/master/R/RBloggersFBXScraper/data.csv"
txt <- getURL(csv.location, cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl"))
df <- read.table(header=TRUE, text=txt, sep=",", stringsAsFactors=FALSE)

It’s then a simple case of subsetting to find posts by a specific author:

find_posts <- function(df, my.name) {
  subset(df, author == my.name)
}

df2 <- find_posts(df, "Tony Breyal")
t(df2[2,])

#                   30
# timestamp         "Wednesday, December 14, 2011 at 10:29pm"
# num.likes         "6 people like this"
# num.comments      "At least 1 comment"
# num.shares        "0"
# posted.by         "R bloggers"
# message           "I love these things  http://www.r-bloggers.com/unshorten-any-url-with-r/"
# embeded.link      "http://www.r-bloggers.com/unshorten-any-url-with-r/"
# embeded.link.text "Introduction\n I was asked by a friend how to find the full final address of an URL \nwhich had been shortened via a shortening service (e.g., Twitter’s t.co,\n Google’s goo.gl, Facebook’s fb.me, dft.ba, bit.ly, TinyURL, tr.im, \nOw.ly, etc.). I replied I had no idea and maybe he should have a look \nover on ..."
# sample.comments   "Kai Feng Chew Yes! It's really cool! I changed a little bit to make it 2 lines to use the shorten function: load(\"unshort.Rdata\") unshort(\"ANY_SHORTEN_URL\") Example:http://cloudst.at/index.php?do=%2Fkafechew%2Fblog%2Funshorten-url-function%2FWednesday, December 14, 2011 at 10:34pm · LikeUnlike ·  1ReplyTony Breyal ‎@Kai\n you might want to use the code from the updated version of the code on \nmy blog because it now handles both https. It won't work with \"http://1.cloudst.at/myeg\" however because that one require the user to be registered (and I'll admit I had not thought of that use case)Thursday, December 15, 2011 at 12:03am · LikeUnlike ·  1Reply"
# rbloggers.link    "http://www.r-bloggers.com/unshorten-any-url-with-r/"
# title             "Unshorten any URL with R"
# first.published   "December 13, 2011"
# author            "Tony Breyal"
# blog.name         " Consistently Infrequent » R"
# blog.link         "http://tonybreyal.wordpress.com/2011/12/13/unshorten-any-url-created-using-url-shortening-services-decode_shortened_url/"
# tags              "dft.ba, R, RCurl, rstats, tinurl, url"

So this tells me that my post entitled “Unshorten any URL with R” got six likes and at least one comment on facebook. Nice. The “sample.comments” field shows what was commented, and that I posted a reply (based on that person’s comment I was able to improve the code and realise that it wouldn’t work with shortened link which requires a user to logged in first). Awesome stuff.

Final Thoughts

So now I have this data I am not quite sure what to do with it. I could do a sorted bar chart with each blog entry on the x-axis and number of facebook likes on the y-axis . I was thinking of doing some sentiment analysis on the sampled comments (I could only scrape visable comments, not the ones you have to press a button to load more for) but I don’t have the time to read up on that type analysis. Maybe in the future

R code: https://raw.github.com/tonybreyal/Blog-Reference-Functions/master/R/RBloggersFBXScraper/rbloggersFBXScraper.R
csv file: https://raw.github.com/tonybreyal/Blog-Reference-Functions/master/R/RBloggersFBXScraper/data.csv