【51CTO專訪】你聽說過Pinterest沒？沒有的話你就Out啦。Pinterest是瀑布流圖片展示形式的發源地，在2010年3月創建。創建之后的9個月內并沒有引起太大的反響，不過從2010年年底開始，忽然發展的一發不可收拾，在今年年中的時候已經超越LinkedIn，成為全美第三大社交網站（前兩位分別是Facebook和Twitter）。今年8月，Pinterest將邀請注冊模式開放為自由注冊模式，推出移動App，準備發起下一波用戶入駐的高峰。

在8月的ArchSummit大會上，來自Pinterest的兩位工程師到場分享了Pinterest的架構擴展歷史。他們分別是Marty Weiner和Evrhet Milam。Marty在2011年1月加入Pinterest，Evrhet在2011年11月加入。對于這樣一個初創企業而言，工程師加入的時間不會很長；但從他們的身上，筆者感受到了他們對Pinterest強烈的歸屬感與認同感。

[[92811]][[92812]]

Marty Weiner & Evrhet Milam

Marty名片上的Title很有意思，叫做Cloud Ninja——云忍者之意。本次對他倆的采訪，就從這個Title的含義開始……

以下，S為51CTO編輯，M是Marty，E是Evrhet。

S：哈嘍，Marty，Evrhet，感謝你們接受51CTO的采訪。首先，Marty，你的名片上用了Cloud Ninja這個稱號。這個稱號很酷！是跟云計算有關系么？

M：是的。我們的工作都在云端控制。而且這個稱號聽起來很好玩：當四處出現各種問題的時候，你四處跳來跳去，感覺就跟忍者差不多。

S：那么，你懂得架構，你懂DevOps，DBA，安全，以及各種為了保持Pinterest穩定運行的技能。但你一開始是程序員。后面那些技能你是從哪里學的？

M：邊干活兒邊學唄。我們基本上都是。我之前做編譯器的，對Web擴展什么的不咋了解。當然，我也做過另一種意義上的擴展，比如讓Java跑的更快什么的，但那個領域不同，跟我現在做的方面層級不同。所以現在我們就是，每個人在各個方向都做點，架構，DevOps，數據庫，產品設計……未來我們工程師多了之后應該會細分一些。Evr，你要不要談談你在做的事情？

E：我關注的領域是用戶參與度的成長，也就是關心用戶有沒有在用我們的產品，有沒有持續的使用，這樣。所以我更多關注功能，關注讓用戶喜愛產品的點，讓他們能夠持續的從我們的產品中得到驚喜。

S：所以，偏產品？

E：是的。這包含很多事情，比如改良推薦的質量和相關性，改善界面，讓用戶的homefeed頁面看起來很酷。

M：同時，每個人都時刻關注網站的狀態，確保在網站出問題的時候可以快速的從備份恢復回來。

S：那你們沒有產品經理嘍？

M：目前還沒有“官方”的產品經理。不過應該很快就有了吧。

S：所有的事情都是工程師在做？

M：哦，我們有一個創始人，他關注整個產品的愿景。我們有設計師。有關注商務方面的人。我們有關注國際化的同事。我們會一起交流，他們告訴我們該往哪些方向進行工程。從某種意義上來說，我們有專注于產品經理的人。

S：那么，工程團隊是怎樣工作的？怎么分工？

E：我們目前有差不多30位工程師。我們將他們分成很多小團隊——越來越多的小團隊。一開始我們有一個完整性團隊，一個增長團隊，然后是一些小團隊，比如Spam團隊，最近有一個Feed質量團隊，還有一個平臺團隊——專門負責API開發和合作伙伴、第三方應用配合的團隊?；旧暇褪呛芏嚓P注不同功能和產品的零碎小團隊。哦，我們還有一個移動團隊。

M：然后我們招聘工程師的時候也會看，可能只有30%是看他的技術能力，看他是不是一個優秀的工程師；還有很重要的30%是文化層面的——我們要確保他們能夠融入我們這個大團隊的文化。他們有激情么？他們有趣么？他們是不是喜歡亂開玩笑？他們是否熱衷于構建、創造東西，讓東西跑起來？如果不是的話，如果他們只是無聊的碼農，那他們就不適合我們的團隊。

S：你們自己面試？

M：是啊，我們要確保團隊文化的傳承，確保新人能夠匹配并延續現有的文化。這也是確保生產力很重要的一點。

S：這樣的團隊組織感覺很另類啊。

M：是的。我覺得一般企業在初創階段都需要設立這樣的門檻。確保你帶進來的人都是有意思的，大家一起構建東西。

S：那，團隊里的每個成員都是啥都懂么？

M：大部分成員對各方面都有一定的了解。他們不一定深入的了解MySQL，但至少知道一些。團隊里面會進來之前做過MySQL的人，比如運維出身的人，他們就可以互相溝通。而這些運維出身的伙計們也需要了解編碼，因為在出問題的時候他們需要有能力做一些修補。

S：那你的專長應該算是在哪個領域呢？

M：我目前在Spam團隊，事實上我們的大部分工作像是救火隊員，哪兒出了問題就撲過去補救。我這個團隊在內部也被叫做BlackUPS，聽起來很酷吧，但實際上我們的工作就是當出了問題，但又沒有具體的團隊負責這個問題的時候，我們就沖上去修補。確保各個方面都正常運轉，這個職責本身就要求我們對各個方面都了解。

E：當然，我們也有專注于數據和推薦這方面的人員，他們相比我們更加專精。他們往往是機器學習或者數據分析相關的博士畢業的。除了他們之外，剩下的人都是萬精油工程師，從服務器擴展，到開發新功能，啥都會。而一般而言我們需要的人也是啥都會才好。

S：有意思。那么，Marty，你在來Pinterest之前是做JVM的。Evh你呢？

E：我之前在Yelp做。也是一個萬精油工程師，做一些信用卡安全防護，數據倉庫，地理位置信息處理方面的工作。

S：你們覺得之前的工作和現在在Pinterest的工作有什么不同么？

M：在Pinterest我學習到很多在大的規模下如何做擴展的知識，以及在這種規模下系統是如何出現問題的。我在這里能夠從比較高的角度觀察整個系統。我學習到如何對大型系統做工程。以前做JVM雖然也有相同的問題，不過JVM和MySQL集群面臨的問題非常不一樣。在編譯器這個級別，你這個系統只有兩種狀態，工作，或者不工作。不是1就是0。而在MySQL的情況下，你會發現這個系統里面有很多技術組件，每個組件的工作狀態是95%！所以你編寫東西的時候，就需要對這種不確定性，這種bug非常的容忍。有的時候一些組件壞掉了，不工作了，但整個系統還在，所以我們的工作之一就是要提升系統的容忍度——在任何不利的環境下仍然能跑著。當然，之前做編譯器方面的經驗還是幫了我很大的忙，至少我知道怎么去debug常規的問題，把它們解決掉，讓它們回到工作狀態。

E：Yelp的情況也差不多吧。都是運行一個用戶量很大的網站。兩個地方的體驗差不多，不過我覺得在Yelp學到的最重要的事情之一就是知道了如何去擴展一個團隊。你要確保代碼是可維護的，有單元測試，有文檔，以確保團隊的其他成員可以繼續我們的工作。Yelp在這方面做的非常棒，我從這個過程中學到很多，比如HR方面的流程，怎么弄來更多的工程師，以及講正確的人放在正確的事上。

M：有一點我覺得很有趣，那就是我們做了很多擴展——Pinterest擴展的非?？?，以至于很多代碼都不成熟。有些代碼簡直是不堪入目。如果你不適當的放慢下來，這個情況就會越來越糟。所以我們招來的一些工程師，有專門負責穩定這個狀態的：他們會重寫大量的代碼，把它們整理干凈了，以準備好接下來的擴展。

S：聽起來不錯。那么，確保你有一個好的團隊是很重要的事情。

M：是的。我們在1月的時候有5個工程師，現在有30個，整個團隊變得更大了。其實很多事情跟你的規模無關，而是跟你運作的多快有關。當你從6個人變成15個人，或者15個到60個，或者100到150個，或者任何一次將團隊規模擴充一倍的動作，你都需要重新定義所有的事情。我們要確保文化的傳承，我們要確保團隊成員們都緊密的團結在一起。我們要確保他們的代碼不會發展的太快，但同時我們也不想設置太多的條條框框——那會減少樂趣，減少“我們在完成一些牛逼事情”的感覺。大公司往往有這樣的問題：你想要改變？先找三層領導簽字吧！那么目前我們仍然處于快速的狀態，但還不至于特別松散。保持樂趣很重要。

S：真棒。那么，你們都做了很長時間的工程師，你們覺得過去這兩三年是不是發生了很多變化？

M：對我而言，絕對是發展非常快速。

S：你覺得這對于開發者們而言是不是有些艱難？

M：怎么說呢，我們招進來的、我們認為合適的人，他們喜歡快速變化的環境。快的環境讓他們感到愉悅。我們這邊有很多人都是從大企業里出逃的，他們覺得那里太慢，所以想回到快速變化的環境中來。

S：他們想要快？

M：是的，他們想要在我們這樣快速移動的初創企業中工作。我見過很多人，他們都說“我受夠了那些管理了！我想寫代碼，完成產品，再寫代碼。我想完成一個產品又一個產品。Pinterest目前還足夠年輕，提供了這樣的機會。所以說，快速是我們的追求。

E：是啊是啊，我加入Pinterest的原因之一就是這個。這里有非常多的開發新產品的機會。回想我剛加入Pinterest的那兩周，可能是我有生以來最緊張的兩周。各種各樣的問題層出不窮，你跑來跑去的，一刻也不能停。我之前在Yelp的兩年從來沒有過這樣的經歷——確實非常不一樣。你會緊張，不停地解決問題，但同時這也帶來極大地滿足感。最后你會覺得，哦，這真有趣！在Pinterest的人都很好，我們像個大家庭，一起睡覺，一起出去玩什么的。我很享受這個過程。

M：我們經常做的一件事就是到看板前面看我們的用戶數量增長的曲線。怎么說呢，無論發生了什么事，你加班到凌晨3點也好，或者隔一兩周來個通宵也好，只要一看到那條曲線，你就會覺得：這是值得的，這是值得的！而且，這個產品本身就是個非常有趣的產品。Pinterest很美——我們有非常好的設計師，有好的愿景，而且它非常簡單。有很多喜愛Pinterest的人，我知道的，我喜歡上Twitter看所有有關Pinterest的推。“我愛死Pinterest了。”或者是那些經過了一天緊張工作的人們說，“好吧，緊張的一天，但至少我還有Pinterest~”看到這些的時候，我真的有很大的滿足感。他們的輕松緩解了我的壓力，因為我知道，我的壓力為很多人帶來了輕松！

S：太棒了。那么，最后一個問題。你們的分享提到了為什么Pinterest做擴展的時候選擇了分片而非集群。那么，這個決策是如何定的？整個流程是怎么樣的，是你們做做調查比較一下就下決定，還是要做測試、數據分析什么的？

M：這個啊，一開始當我們只有2、3個人的時候，很多情況下其實就是“好吧，我們希望這能行”，就去做了。我們當時沒有足夠的條件去測試，一一篩選那些不合適的技術。而當我們知道某項方案可行的時候，我們會做一個拆分，選擇我們認為可能給我們最佳結果的方式去做。最后它成功運行，其實是有一些運氣的成分。當然，現在我們有能力做一些測試了，不過更多的情況下我們還是直接構建了系統，推送到部分的服務器上，看是不是會出現什么問題，再做進一步動作。比如我們的關注服務，一開始會推送給10%的用戶，看看運行的情況，發現有些問題，解決掉它們。然后再推送到20%，好吧又出現了新問題，那么解決它們，再前進。所以，就這樣直接在生產環境中進行測試，直到我們推送到100%，這就完成啦。控制起來并不困難。你可以推送模擬的負載來測試系統，觀察系統是如何反應的，但是模擬負載與真實的負載從來都不同。從來都不同。

E：同時，這也取決于你的團隊氛圍。當有一些新的想法的時候，工程師們會討論它們。我們對這新想法感到興奮么？好的想法往往能引發更多的討論，工程師們也會開始嘗試將新想法帶進來，測試，部署。總之，是工程師對新想法的興奮感，這才能為我們遇到的問題帶來好的解決方案。

S：有領導者嗎？

M：我們有一個Head of Engineers。然后，每個小團隊都有一個小組長，不過小組長主要的工作不是管理，而是帶領團隊一起編碼。當然，還有我們的創始人。兩個創始人默認在所有人之上的。

S：好的。那么，十分感謝Marty和Evrhet接受我們的采訪！

第二頁是英文采訪實錄。

#p# 

s: hi Marty, Evrhet, thanks for taking our interview!

You call yourself a cloud ninja. Does this have anything to do with cloud computing?

M: yes. When we do our work, the control system is all on the cloud. And it just sounds funny. When there are lots of problems going on there, you have to be a little bit ninja.

S: you know architect, devops, dba, security, and everything else to keep pinterest running. And you started off as a coder. When did you learn all these things?

M: on the job. both of us. i was a compiler guy, and i didn't do scaling for web. i did big scaling on the other side of the game, like making java run faster, but that's on the other side. i didn't do things quite on that level. so that's the way we are going. and we do a little bit of everything, architect, devops, db guy, product feature...eventually we get segment as we get more engineers. i do spam. Evr, how about you talk what you do?

e: i'm focused on engagement growth, so trying to make sure more users are using our product, and continue to use it, day by day. so just features, making people enjoy the product, help them find contents they are interested in.

s: so a lot on the product side?

e: mine is a lot more product focused, so we do things to make sure you have good recommendations, they are pretty to look at, make users happy when seeing their homefeed.

m: every eyeball still keeps an eye on the website, in case it breaks, get a backup and get it back running.

s: do you have a product manager?

m: not yet. no official individual product manager yet, but i think we'll get one.

s: so just the engineers doing everything?

m: oh we have a founder who is focused on the vision. we have some great designers. and people focused on the business. we have people focused on internalization, and help us understand what we need to engineer that. so we have product management in implicit way, but not a specific product manager yet.

s: so how is the engineering team in pinterest like? how do you distribute work?

e: i think we have over 30 engineers. and we are splitting them into more and more teams. we have originally an integrity team, and a growth team, and then we had even smaller teams, like the spam team, recently a team related to feed quality, and we have a platform team, that just worries about the apis, and making the platform available for partners and third-party apps. so we just have a lot of small teams flying around different features and products we have. and we have a mobile team, too.

m: and in terms of the type of people that we look for, 30% of our interview is on technical aspects, to make sure they are good engineers, but a 30% is on cultural - how good they can fit into our culture. are they motivated, are they fun, and they like to joke around. do they want to build things and make them run, and create. if they are not like that, if they can only write code but they are not fun, we won't hire them.

s: so you interviewed them?

m: yes, we all interview the new people coming in, that's the culture we want going forward, so that to make sure people coming in match the current culture, in the way that you like, i think it's productive.

s: that's quite unique.

m: yes, i think that barrier is common at the early stage of companies. make sure the people bring in a having lots of fun. build things.

s: so do all the members of the teams know everything?

m: most people know a little bit about every piece, but they may not know, day to day, about mysql, we are a little bit segmented. but a lot of people we hire have just done different pieces, some of them know well about mysql if that's what they previously did. but people who know mysql are most probably the ops guys, and they need to know code, need to know how to fix things when it breaks.

s: are you regarded as specialized in certain fields, then?

m: i'm working on spam right now, we tend to jump on whatever needs to be fixed now and then. i'm actually on the team that we call "blackups", it sounds cool, but what it does is if there is something broken and there is not a specific team to fix it, we'll go and fix it. we are the people that make sure everything is fluid and working. so this role requires us to know a little bit about everything.

e: we do have some people who are focused on data and recommendations, they are a bit more specialized than the rest of us. they have knowledge of phds, on machine learning or things like that, or analytical knowledge. most of us are like generalist engineers, and can come to deal with scaling servers, add a feature, and things like that. usually we need people who can jump onto anything.

s: that's interesting. so marty, before you joined pinterest you were doing jvm. how about eve?

e: i was at yelp. a generalist engineer that was doing credit cards protection, data warehousing, and locations.

s: so what difference do you think there is between working in pinterest and working in your previous post?

m: i think i really learned how to scale things on a larger scale, how things break down, i get to see things from a top level. i really learned how to engineer very large systems. it's the same problem, but also a very different problem from scaling mysql. from a compiler level, your system(?) either work or doesn't work, it's 1 or 0. for mysql, you find yourself surrounded by technology that works 95%. you have to write things that are very tolerant to that, tolerant to the bugs. some of them don't work, some of them break away, so part of our job is to make things tolerant - make them work no matter what is happening. i think the compiler work helped a lot for me, just for debugging regular problems, beating them down to the ground to make them work.

e: yelp does pretty much the same thing, running a website that has a lot of users. so i have a lot of similar experience that i have now. but i think yelp has a lot of knowledge in scaling a team. making sure the code is maintainable, has unit tests, is documented, so that the rest of the team can carry on with the code. yelp had a good job in doing that, and i learned a lot of these processes, like hr processes, getting more engineers, getting the right people for the job.

m: one thing i think is interesting is that, it is just two of us for wild scaling, when pinterest is scaling fast, a lot of code is not matured. some code there is just really nasty, and if you don't slow down a bit, it will just become worse and worse. so some engineers we brought in, their work is to stablize, so they come in and rework a lot of the codes to make them cleaner, and more scalable for later.

s: cool. so the important thing is to get the right team.

m: yes. in january we had 5 engineers, and now we have 30. so we are getting bigger in size. it actually doesn't matter what scale you are at, but about how fast things move. so when we moved from 6 to 15 engineers, and 15 to 60, or 100 to 150, or any doubling the size, you have to rework everything that you used to. so we have to make sure the culture matches, and we have to make sure we have properly related our team. we have to make sure their coding doesn't fly too fast, but at the same time we don't want to place too many rules around the place, that will slow down the fun, slow down the feeling that you are getting something done. bigger companies have this problem: i want to make a change, i have to go three levels for approval. right now we are still fast, but not super loose. keep it fun.

s: cool. so you have been engineers for quite a long time, do you think things have changed a lot in these 2-3 years?

m: it certainly is moving fast for me.

s: and do you think this has made things more difficult for developers?

m: i think we get the right people who likes things to be fast moving. it makes things better for them. we get a lot of people who come from big big companies that have slowed down. and they want to come back to the fast.

s: so they want the fast changes?

m: yes. they want to be in small startups that move fast. we've got a lot of people who say "i'm sick of all the management. i want to make code and move. i want to make another product and move. pinterest is still young enough so they can get that opportunity. so fast is what we thrive on.

e: oh yes, that's one of the reasons i joined pinterest. there are so many opportunities to work on new products. Right after I joined, it was probably the most stressful two weeks of my life. Thing were breaking all the time, you keep going back to a fixing mode - i've never experienced that before in the two years at yelp. it's really different. it can be stressful, keep solving problems, but it is really satisfying, very very fun at the end of the day. so in pinterest we have all the right people, like families, sleep and hangout together, i really enjoyed it.

m: one thing is that we kept watching that graph of growing number of users that keeps rising. so no matter what happens, whether we worked up till 3am or even all night every week or the other week, as long as you saw that graph, it's like, oh, it's worth it, it's worth it~ and also it's a fun product to build. it's a really beautiful product, we get really good designers, visionaries, and it's really simple. a lot of people had so much fun with it, and i like going on to twitter and reading all these people saying "i love pinterest". a lot of people would go in the end of stressful days and say, oh at least i've got my pinterest and my key. reading those is really satisfying. you know a lot of people is relaxing, you know, my stress! at least my stress is going to relax somebody:)

s: cool. so last question. your session mentioned choosing sharding over clustering. so who make those decisions? what's the process like? do you just go ahead and decide, or do some data, testing before you went for it?

m: so early on, when there were only 2 or 3 of us, it was pretty much like "we hope this will work", because we just didn't have the room to test and bring up a lot of technologies that didn't work. and we knew what was working, so we sort of extract it, and hope for the best thing we got. and in a way we got lucky as it turned out to be working. now we do have the capacity to do a bit of testing, but we are still kind of build the system and start push it to a little bit of entity(?) and see how it does, and if there are problems, fix them. For the follower service we started off pushing 10% and see, ok we've found a few problems, fix those. push 20%, ok a few more problems, and so on. so we have this bring things to production, but just push some things to it. and when it's 100%, you are done. so in a way we are kind of test in production, but in control. and you can always push loads to the system, you can see how the system responds, but it never ever matches the real load.

e: also i think it's up to the team you have. when new ideas come up, engineers are going to talk about it. are they excited about it? good ideas tend to have lots of chats, and people will start bringing them in. test them. deploy them. they have to be excited about the new idea - good solutions to the problems we have.

s: so do you have some kind of a leader?

m: we have a head of engineers. and then, every team has a team leader who is very down into the code - he is not managing, just code along and lead the teams. and we have a founder, two founders who are automatically on top of everything.