Wednesday, February 23, 2005

On Monday, at 10.14 pm UTC, the Wikimedia server cluster experienced a total power failure, taking down Wikipedia, Wikinews, and all other Wikimedia websites.

The servers are housed in a colocation facility (colo) in Tampa, Florida, USA. They occupy two racks, with each rack receiving electricity from two independent supplies. However, both supplies have circuit-breakers in them, and both opened at the same time, leading to a total power failure. All computers immediately went down. It’s normal for fire safety regulations to prohibit uninterruptible power supplies in colos, with the colo providing its own UPS and generator instead. The circuit breakers were on the computer side of this emergency power system, so none of the computers continued to receive power to survive the breaker trip or shut down safely.

The actual reason why the circuit breakers tripped is currently unknown.

When power was restored, it was discovered that most of the MySQL databases that store the data which makes up Wikipedia et al had been corrupted. The main database and the four slaves had all damaged the data on their hard disk drives beyond the ability of the auto-correction to repair. Only one copy survived safely, on a machine that is used for report generation and maintenance tasks, which remained 31 hours backlogged while catching up after an unusually heavy update load during the previous week.

Volunteer Wikimedia engineers worked through the night rebuilding the databases from the sole good copy onto the other servers. The Wikipedia database is over 180Gb in size, making the copying process last 1.5 – 2 hours for every server it was performed upon.

Regular back-ups of the database of Wikipedia projects are maintained – the encyclopedia in its entirety was not at risk. The last database download was made on February 9; all edits since then could only have been laboriously rebuilt from logs and recovered from the damaged database requiring much more time and effort.

Limited read-only service was established late Tuesday afternoon, with editing becoming possible 24 hours after the power failure. Final repairs continue now, as well as upgrades to prevent similar issues in the future. Server-intensive features, such as categories and ‘watchlists’ that display recent changes to selected articles to registered users, remain disabled to ease the load on the recovering systems.

The process which led to the damage originated with the operating system, disk controllers, or hard drives failing to flush the data correctly.

If the power to a database server is cut mid-write, the database may be corrupted and unreadable, however the operating system, hardware, and software are designed to make this very unlikely. In a previous incident in 2004 power was also lost to a server but the database was undamaged.

To avoid such damage, each database server saves a copy of an edit to be applied to the database on a separate storage system before making the actual update to the database itself. This so-called ‘write-ahead logging’ should ensure that in the event of a system crash, the database can be rebuilt from a ‘last-good’ state by replaying the edits saved in the log.

Earlier this year popular blogging site LiveJournal suffered a similar power failure when another customer at their colocation facility pressed an Emergency Power Off button, intended for use only by firefighters. The company suffered database corruption similar to that seen at Wikimedia.

LiveJournal are now fitting UPS to their servers to ensure that they have time to shut down safely in the event of a power failure. Wikimedia was said to be investigating the possibility of fitting similar equipment at the time of this failure.

Several pundits have suggested that the use of another database, such as the proprietary database Oracle or the free PostgreSQL, would have avoided the database corruption seen at the server cluster. A post-mortem of the incident show the failure was in the operating system, or the hardware, or some combination of the two. LiveJournal, which also uses MySQL, reported similar database corruption after their power cut.

The Wikimedia foundation only allows the use of free software on its systems, and future versions of the Mediawiki software will support the PostgreSQL database.

Users are reminded that during times of system failure or excessive demand, they can still search Wikipedia using Google. The articles may be viewed using Google’s cache.

Saturday, May 23, 2009

British Airways (BA), the flag carrier for the United Kingdom, announced record losses today. For the first quarter of the year ending March 31, the airline reported losses of £401 million before tax, despite revenues which rose to £9 billion over the previous year. The company’s finances have suffered from rising fuel costs and a weak pound. The loss also takes into account redundancy-related costs of £78 million: The company laid off 2,500 workers last year.

BA’s costs soared due to an unprecedented fuel bill of £3 billion following a 44.5% increase in the price of oil last year. The recent drop in oil prices has led the airline to announce an expected saving of £400 million in fuel by next year.

Company CEO Willie Walsh announced that employees will be offered the option of taking unpaid leave or working part time in order to ease the financial crisis. In order to set an example, he said that he himself would forego pay for the month of July, describing his move as “no stunt,” and added that he “saw no signs of recovery anywhere.”

Part of BA’s troubles over the last year could be due to the company’s dependence on the premium travel market. The company saw a drop of 13% in ‘Club’ and ‘First Class’ passengers. With two new aircraft on the way to operate a Club-only service from London, the company may struggle to fill the flights.

Thursday, May 27, 2010

After reports that US President Barack Obama has fired the director of the Minerals Management Service, Elizabeth “Liz” Birnbaum, the Interior Department revealed that she instead resigned “on her own volition”. The resignation occurred amidst growing criticism of the federal government’s response to the Deepwater Horizon drilling rig explosion and the agency’s oversight over offshore drilling. 

Birnbaum, who became director in June 2009, was expected to testify before a subcommittee of the House of Representatives today with Interior Secretary Ken Salazar. She was not present when Salazar began to speak, however. Salazar said in a statement: “She is a good public servant. She resigned today on her own terms and on her own volition. I thank her for her service and wish her the very best.”

News agencies had reported that Birnbaum was forced out of office, and Obama was expected to officially announce the supposed firing later today in a news conference along with discussing an Interior Department report on the explosion. Birnbaum observed in her letter of resignation that Salazar “will be requiring three new leaders for the Office Natural Resources Revenue, the Bureau of Energy Management and the Bureau of Safety and Environmental Enforcement” as the entire Minerals Management Service is reorganized.

Sunday, January 31, 2010

China has stated that they will cease all military exchanges with the United States in response to a US$6.4 billion (NT$204 billion) weapons deal with Taiwan going ahead. He Yafei, Chinese Vice-Foreign Minister, said that there would be a “serious negative impact” with regards to the relationship between the US and China.

Taiwan and mainland China have been ruled by separate governments since 1949, and neither recognize each other’s independence. In an effort to regain control of Taiwan, the government in Beijing has been open to running the island like Hong Kong, in a “one country, two systems” manner, but people in Taiwan support the status quo. The Chinese defense ministry said that they had “decided to suspend planned mutual military visits”. They “strongly demand” that the United States “respect the Chinese side’s interests,” calling for the proposal to be rejected.

China summoned the US Ambassador to China, Jon Huntsman, to warn the US Government about the consequences, and to urge for cancellation of the deal. The Pentagon proposed the deal, that was first promised during the later days of the Bush administration, to Congress earlier.

Yafei issued a statement on the Foreign Ministry website stating that “the United States’ announcement of the planned weapons sales to Taiwan will have a seriously negative impact on many important areas of exchanges and co-operation between the two countries.”

However, the US is defending itself. “Such sales contribute to maintaining security and stability across the Taiwan Strait”, Laura Tischler, spokeswoman for the US State Department, said.

Beijing is threatening to review co-operation on major issues and impose sanctions on companies selling arms, even though US companies are banned from selling arms to China as a result of the Tiananmen Square massacre in 1989, so it is not clear what the effect will be. A Chinese defence ministry spokesman said that the sanctions reflected the “severe harm” the deal would have. A foreign ministry spokesman stated that there would be “repercussions that neither side wishes to see” if the deal went ahead.

Taiwan, one of Asia’s few functioning democracies, welcomed the proposal, with Taiwanese president Ma Ying-jeou being quoted as saying “It will let Taiwan feel more confident and secure so we can have more interactions with China.”

The arms deal includes UH-60 Black Hawk helicopters, communications equipment and MIM-104 Patriot missiles, but does not include the F-16 fighter jets that Taiwan’s military were looking for. The US is legally obliged to supply Taiwan with defensive weapons under the Taiwan Relations Act 1979.

In 2003, an unnamed US briefing stated that “[the US] would have to get involved if China tried to use coercion or force to unilaterally change the status of Taiwan”.

Friday, January 25, 2008

B2B Trade Area of Taipei Game Show, criticized by trade buyers last year, but accompanied with 2008 Taiwan Digital Content Forum, moved to the second floor at Taipei World Trade Center for world-wide participants with a better exchange atmosphere this year.

Not only local OBMs (Softstar Entertainment, Soft-World International Corp., International Games System Corp., …, etc.) but also companies from New Zealand, Canada, Japan, Hong Kong, and South Korea showcased different specialists with multiple styles. Especially on South Korea, participated members from G? Trade Show (Game Show & Trade, All-Round, aka Gstar) showcased gaming industry of South Korea and the G? upcoming at this November with brochures.

In the 2-days Digital Content Forum, world-class experts not only shared industry experiences, members from Taiwan Gaming Industry Association also discussed and forecasted marketing models for gaming industry. With participations from governmental, industrial, and academical executives world-wide, this forum helps them gained precious experiences of digital content industry from several countries.

According to the Taipei Computer Association, the show and forum organizer, the digital content industry in Taiwan was apparently grown up recent years as Minister of Economic Affairs of the Republic of China Steve Ruey-long Chen said at Opening Ceremony yesterday. Without R&Ds from cyber-gaming, and basic conceptions from policies and copyright issues, this (digital content) industry will be fallen down in Taiwan. If this industry wanted to be grown up in sustainability, gaming OBMs in Taiwan should independently produce different and unique games and change market style to market brands and games to the world.

Friday, January 25, 2008

B2B Trade Area of Taipei Game Show, criticized by trade buyers last year, but accompanied with 2008 Taiwan Digital Content Forum, moved to the second floor at Taipei World Trade Center for world-wide participants with a better exchange atmosphere this year.

Not only local OBMs (Softstar Entertainment, Soft-World International Corp., International Games System Corp., …, etc.) but also companies from New Zealand, Canada, Japan, Hong Kong, and South Korea showcased different specialists with multiple styles. Especially on South Korea, participated members from G? Trade Show (Game Show & Trade, All-Round, aka Gstar) showcased gaming industry of South Korea and the G? upcoming at this November with brochures.

In the 2-days Digital Content Forum, world-class experts not only shared industry experiences, members from Taiwan Gaming Industry Association also discussed and forecasted marketing models for gaming industry. With participations from governmental, industrial, and academical executives world-wide, this forum helps them gained precious experiences of digital content industry from several countries.

According to the Taipei Computer Association, the show and forum organizer, the digital content industry in Taiwan was apparently grown up recent years as Minister of Economic Affairs of the Republic of China Steve Ruey-long Chen said at Opening Ceremony yesterday. Without R&Ds from cyber-gaming, and basic conceptions from policies and copyright issues, this (digital content) industry will be fallen down in Taiwan. If this industry wanted to be grown up in sustainability, gaming OBMs in Taiwan should independently produce different and unique games and change market style to market brands and games to the world.