In the coming months we will launch ChinesePod Version 3. It will bring about some fairly big changes. These changes are based on the tremendous flow of feedback that we’ve had from you, the users. In preparation for that new iteration, we’ll write a series of posts outlining the changes, starting with this one.
The biggest overall change in Version 3 is to dramatically increase our commitment to server reliability. The feedback indicates that this is the single biggest problem that you have been experiencing. We believe it supersedes all other changes, since without a fully reliable server environment, the others don’t ultimately matter!
Server Environment
We currently host in a multi-server environment in the US to maximize availability for both our US and International audience. However, the rapidly going popularity of ChinesePod has exceeded our expectations and caught us a little unprepared. As a result, we currently don’t have the server redundancy necessary to avoid downtime because of excessive server load or hardware failure. This will change as we make the necessary investments to build in this redundancy.
MP3 Access
Every day we serve several hundred gigabytes of data (primarily MP3’s) to the global audience of ChinesePod users. While bandwidth cost is something all podcasters need to manage, there are services such as Libsyn that offer flat-rate hosting charges which reduce these costs. These companies can offer this pricing by throttling download speeds - speed up some downloads, slow down others - in an effort to control the total amount downloaded at one time. This is very attractive to many podcasters, but the downside is access during peak times is slow to the user.
To solve this problem we will move all MP3 hosting to Amazon S3. This service is built on Amazon’s existing infrastructure and provides the same levels of access and reliability.
Another part of this problem has been our high MP3 bitrate of 128 kbps which preserves quality, but maximizes file size. While we will continue to offer free podcast lessons on ChinesePod, we will also begin to offer more versions of the file to let each user make their own choice between file size and download speed. In addtion, we will provide a separate audio file that will only contain the core dialog without the explanations and host banter.
- 64 kbps episode version - will be used for public RSS feed, the audio player in the lesson post and in the flash player for syndication
- 128 kbps episode version - will be available in private RSS feeds and right-click download on the lesson pages
- 128 kbps dialog version - will be an additional option in the audio player in the lesson post and available for right-click download on the lesson pages
- 32 kbps episode version - will be available as an option in private RSS feeds for those who want to maximize download speed or minimize data fees if downloading over GPRS/3G
In addition, all word and sentence audio in lesson review will be encoded at 32 kbps per second and also hosted at Amazon S3 to maximize accessibility.
At the end of the day this is all ‘plumbing’ and should be invisible to the user. It is our responsibility to make it that way. We appreciate your patience.
中文 Chinese
hanyu_xuesheng Says:
February 23rd, 2007 at 7:57 pm
Your plans make sense. Thank you for listening to your customers.
Lantian Says:
February 23rd, 2007 at 10:33 pm
SERVES UP - I don’t use iTunes much to get my Cpod mp3s so I’m not sure if the mp3s there also weren’t available when the Cpod site would go down. Taking a quick look at Amazon S3 they offer a service committment of 99.99% availablity without any single points of failure. That’s about an hour of downtime per year. Which is certainly adequate for Chinesepod.
My concern is about the Wordpress application and how this change would affect it. Does this application port over to a distributed server environment? What backend database does it use? I’m guessing that Cpod’s V3 is actually a new custom-built application. Is that right?
I’m pretty sure that after the switch to S3 and working out the kinks that the system will be very reliable for the global audience.
But Cpod has had a lot of growing pains (from an observer on the outside) with the actual implementation and rollout of changes, not so much the backend infrastructure. If you are down two or three days with a switch over, it’s a huge impact to your current now rather large audience.
It is not easy to test a distributed application on a single server, nor is it easy to implement to the multinode environment. Will you be able to test V3 on Amazon’s S3 before retiring the current Cpod application and host? Do you have a local in the Factory multi-server environment to test? Will Cpod V3 have automated and schedulable functions to handle all the various mp3 versions? The challenge of switching over the domain name in a timely fashion globally also makes it tough.
Wow, anyway, just thinking out loud. You guys are really great, onwards and upwards!!
海�? / Henning Says:
February 23rd, 2007 at 11:39 pm
The luxury pains of having a growing business…
Those measures sound highly promising!
Especially the dialogue-only-versions are an interesting addition to your service with a visible extra value.
Maybe you could further curb bandwidth consumption by putting the links to the high-quality versions in the premium-area?
Lantian Says:
February 24th, 2007 at 2:09 pm
Improving Chinesepod Availabiltity
I. A review of a Cpod high-availability infrastructure
————————
1. User: Firefox, Internet Explorer, Safari, iTunes
2. Internet: North American audience, intl. audience, China audience
3. Applications: Wordpress, mp3 RSS loader, other
4. High-Availability middleware:
5. Server OS: Apache
6. Database: MySQL
7. Hardware: hosting service
———————–
II. Possible Changes
1. User: no changes, insure new application works well with all browsers via in-house testing on various browser versions and iTunes.
2. Internet: Does S3 offer a service level agreement for international bandwidth? How will Cpod support China mainland customers? Is a mirrored site within China feasible? Will the Wordpress or V3 applications be hosted by another service? Will there be some sort of mirroring for intl, and mainland China?
3. Applications: Is Wordpress tune-able for HA? Is the new application and Cpod V3 tested for a HA-clustered environment? Is the new site separating out static content on Wordpress with dynamic content on S3? What db will store the mp3s?
4. HA Middleware: Is this available on S3, what is within cpod’s ability to administer? How will this middleware control applications on different hosts?
5. Server OS: What is available to Cpod to customize? It’s unlikely S3 gives client admin control over a whole server. Is S3 going to be tuned to Wordpress, or the mp3 serving application, and Cpod needs?
Tuning Apache OS for Wordpress and high traffic
http://www.howtoforge.com/conf.....erformance
6. Database: Will this be done?
Implementing a high-availability cluster for MySQL database
http://www.oreillynet.com/pub/.....uster.html
http://developer.amazonwebserv.....;tstart=-1
7. Other: Are there plans to separate static content from dynamic content for Cpod website. There’s no reason the whole site has to go down if one area fails or due to bandwidth. It’s unlikely the blog, comment sections become so inundated with client requests. Can the links to mp3s be links to a separate server hosting of dynamic/mp3 content? This is exactly the strategy with the move to Amazon’s S3 right?
8. PIZZA: Is there enough pizza money and local delivery?
New server environment « The ChinesePod Blog with Ken Carroll Says:
February 25th, 2007 at 12:57 pm
[...] Horkoff posted about an important issue at the Praxis blog during the holiday - namely the question of server reliability. As you know [...]
Charles Says:
February 25th, 2007 at 2:18 pm
I thought that a move to S3 might end up happening with chinesePod dueto the sheer cost saving other web2.0 companies are reporting. Unfortunately being in mainland china this isn’t such good news for me. I’ve always found that Amazon s3 sites to be excruciatingly slow here. Amazon.com can take up to 2mins to load… a quick check just then clocked it in at 72secs for the front page and another 49 sec to bring up the books section under browse. I have no idea why this is Amazon always loaded very fast back in Australia and now it would have to be the slowest site that I regularly visit.
Lantian Says:
February 25th, 2007 at 8:18 pm
Mainland Mirror - Why can’t Cpod have a Asia mirror? At least for the mp3s.
The best electronic Chinese-English dictionary has one.
http://pleco.com/
January 23 , 2007
We’ve just added a mirror server in Asia for faster file downloads from China, HK, and Taiwan.
Server upgrade « Chinees leren Says:
February 28th, 2007 at 5:06 pm
[...] moeten leiden tot een stabielere en ook snellere serveromgeving. Voor meer details moet je dit blog er maar eens op nalezen. De werkzaamheden beginnen vanacht om 3 uur onze tijd (Brussel, Amsterdam). [...]
Usability Improvements « The ChinesePod Blog with Ken Carroll Says:
March 12th, 2007 at 9:21 am
[...] coming to the next version 3 of the ChinesePod service. Continuing on from our previous post - Access Improvements - today we discuss some of the new usability [...]