Over the last year or so a bunch of presumptuous European sysadmins and developers, joined by some of their American brethren and even a couple of us antipodeans (there are others too!) have been talking about a concept called DevOps.  DevOps is the merger of the realms of development and operations (and if truth be told elements of product management,QA, and *winces* even sales should be thrown into the mix too).

過去一年以來，一批來自歐美的、不墨守陳規的系統管理員和開發人員一直在談論一個新概念：DevOps。DevOps 就是開發（Development） 和運維（Operations）這兩個領域的合并。（如果沒錯的話，DevOps還包括產品管理、QA、*winces* 甚至銷售等領域）

The Broken

脫節(The Broken)

So … why should we merge or bring together the two realms?  Well there are lots of reasons but first and foremost because what we’re doing now is broken.  Really, really broken.  In many shops the relationship between development (or engineering) and operations is dysfunctional to the point of occasional toxicity.

那么……為什么要合并這兩個領域？原因很多，但首要原因是：我們目前的工作流程是脫節的。絕對的脫節。很多公司的開發部門和運維部門之間存在的深刻矛盾，其實就是這個“脫節”造成的。（意譯，求斧正）

Here’s an example I think everyone will be at least partially familiar with: the minefield that is project to production software deployment.  Curse along as I explain.

下面是一個大家都基本熟悉的例子：部署軟件產品。

Development builds an application, the new hotness which promises customers all the whizz-bang features and will make the company millions.  It is built using cutting edge technology and a brand new platform and it has got to be delivered right now.  Development cuts code like crazy and gets the product ready for market ahead of schedule.  They throw their masterpiece over the fence to Operations to implement and dash off to the pub for the wrap party.

開發部門要開發一款新產品。這款產品要使用最新最炫的技術，來保證客戶的所有花俏的需求，從而給公司帶來百萬美元的利潤。這款產品被要求使用最新的技術和運行平臺，還得馬上交付。于是開發部門沒日沒夜的加班、趕代碼(cuts code like crazy），終于如期完成了任務。然后他們把自己的“杰作”一股腦的甩給了運維部門，后者還沒能完全接手，前者已經迫不及待的開始了慶功會。

Operations catches the deployment and is filled with horror.

接到產品后，運維部門每個人的心中都充滿了恐懼。

The Operations team summarises their horror and says one or more of:

下面就是運維部門的恐懼之源：（ {A.B.C} 表示 A 或 B 或 C 之一 )

    * The wonder application won’t run on our infrastructure because {it’s too old, it doesn’t have capacity, we don’t support that version}

* 這款優秀的產品在目前的底層平臺上無法運行，因為這個平臺{太古老了，空間不足，不支持某某版本}

    * The architecture of the application doesn’t match our { storage, network, deployment, security } model

* 這款產品的體系結構跟我們的{存儲，網絡，部署，安全}模型不匹配。

    * We weren’t consulted about the { reporting, security, monitoring, backup, provisioning } and it can’t be “productionised”.

* 這款產品的{ 報告，安全，監視，備份，服務提供} 我們搞不懂 ，所以沒法把它做成實際可用的產品。
 

But Operations persevere and install the new hotness – cursing and bitching throughout.  Sadly, after forcing the application onto infrastructure and bending and twisting the architecture to get it running, the performance of the new application can be summed up as “epic fail”.

盡管伴隨著不絕于耳的抱怨和咒罵，運維部門最終還是把這款產品安裝好了。不幸的是，由于做了很多蹩腳的修改和不合理的強迫式運行，這款產品的性能最后被歸結為：終極失敗（Epic Fail）。

Operations sighs and starts logging problems and passing issues back to the Development team.  Their responses generally come from the following pool:

于是非常沮喪的運維部門開始記錄各種問題，源源不斷的給開發部門提Issue。而開發部門的回應基本上都是：

    * It’s not our fault – our code is perfect – it’s just been poorly implemented

* 這不是我們的錯 —— 我們的代碼非常完美——而是（運維部門的）部署做的太差勁了。

    * Operations are stupid and don’t understand the new hotness – why can’t they implement the cutting edge technology? Why are they so backward?

* 運維部門比較笨，他們不懂新技術—— 為什么他們沒法實現最新的技術呢？為什么他們這么落伍呢？

    * It runs fine on my machine…

* 在我的機器上運行的沒問題啊……

The interactions between teams quickly becomes a toxic blame storm. The customers (and by extension the shareholders, investors and management) then become the losers.  The loop gets closed with the company losing bucket loads of money and everyone losing their jobs.  EPIC and FAIL.

兩個部門之間的交流很快變成了一場暴風驟雨。客戶（以及股東、投資方和管理層）則成了蒙受損失的失敗方。最終公司損失了無數的金錢，大家也都失業了。終極的失敗。

What’s different about DevOps?

DevOps 又有啥不同？它有什么好處？

DevOps is all about trying to avoid that epic failure and working smarter and more efficiently at the same time. It is a framework of ideas and principles designed to foster cooperation, learning and coordination between development and operational groups. In a DevOps environment, developers and sysadmins build relationships, processes, and tools that allow them to better interact and ultimately better service the customer.

DevOps 就是想方設法的避免這種“終極失敗”，同時讓大家用更聰明更有效的方式去工作。它是一種框架，包含了很多優秀想法和原則，它鼓勵開發部門和運維部門通力合作。在DevOps環境中，開發人員和系統管理員會構建一些關系、流程和工具，從而更好的與客戶互動，最終提供更好的服務。

DevOps is also more than just software deployment – it’s a whole new way of thinking about cooperation and coordination between the people who make the software and the people who run it.  Areas like automation, monitoring, capacity planning & performance, backup & recovery, security, networking and provisioning can all benefit from using a DevOps model to enhance the nature and quality of interactions between development and operations teams.

DevOps 也不僅僅是一種軟件的部署方法。它通過一種全新的方式，來思考如何讓軟件的作者（開發部門）和運營者（運營部門）進行合作與協同。使用了DevOps模型之后，會使兩個部門更好的交互，使兩者的關系得到改善，從而讓很多領域從中受益，例如：自動化、監視、能力規劃和性能、備份與恢復、安全、網絡以及服務提供（provisioning）等等。

Everyone in the DevOps community has a slightly different take on “What is DevOps?”  We all bring different experiences and focuses to the problem space.  I personally see DevOps as having four quadrants:

“對于DevOps是什么？” 這個問題，DevOps社區中的每個人的回答都不盡相同。因為我們的工作經驗不同，關注的問題也不同。就我個人而言，DevOps分成四大部分：

Simplicity

簡單

KISS is King and in that vein this section is simple too. Design simple, repeatable, and reusable solutions. Simplicity saves documentation, training, and support time.  Simplicity increases the speed of communication, avoids confusion, and helps reduces the risk of development and operational errors.  Simplicity gets you to the pub faster.

KISS（Keep it Simple and Stupid，簡單就是美）原則是最重要的。所以本段文字也很簡單。我們要盡量提供簡單、可重用的解決方案。“簡單”節約了書寫文檔、培訓和提供支持的時間。“簡單”增加了溝通的速度、避免混淆、減少了開發和運維出錯時的風險。“簡單”讓人更快的發布產品。

Relationships

部門之間關系

Engage early, engage often. Development teams need to embed operations people into their project and development life cycles.  Invite operational people to your scrum or development meetings.  Share ideas and information about product plans and new technologies. Gather operational requirements when gathering functional ones. As a project progresses test deployment, backup, monitoring, security and configuration management as well as application functionality.  The more issues you fix during the project the less issues you expose your customers to when the application is live.  Educate operations people about the applications architecture and the code base. The more information operations people can feed you about a problem with the code the less trouble-shooting you need to perform and the faster the problem can be fixed.

早參與，多參與。對于開發人員，要讓運維人員常駐到開發部門，全程參與開發流程。邀請運維人員參與你的Scrum或者開發會議，與他們分享項目計劃、分享新技術的點子和心得。搜集功能性需求（指開發人員用到的需求）的同時也要搜集運維方面的需求。把對于“發布、備份、監控、安全、配置管理和系統功能”的測試作為一項獨立的項目流程。軟件產品在開發時解決的問題越多，那么在使用時暴露給用戶的問題就越少。給運維人員做培訓，讓他們弄清楚項目的體系結構和核心代碼。如果運維人員在反饋bug時提供的信息越多，那么你花在排查問題（trouble-shooting) 的時間就越少，這個bug也就會更快的被解決掉。

Operations people need to bring development people into the problem and change management space. Invite developers into your team meetings. Share your roadmaps and upgrade plans.  Understand where future development is heading to better ensure infrastructure deployments match product requirements.  Developers also bring skills, knowledge and tools that can help make your environment easier to manage, more efficient and cleaner. Learn to code or if you’re a hack-n-slash systems programmer like me then learn to code better. 

Concepts like building tools with APIs rather than closed interfaces, distributed version control, test driven development, and methodologies like Agile Development, Kanban and Scrum can revolutionise operational practises in the same way they’ve changed the way code is cut.

對于運維人員，在遇到問題時需要把開發人員加進來，大家一起解決問題。邀請開發人員參與你們的會議，分享項目進度(roadmaps)，并且共同修訂工作計劃。運維人員一定要了解開發部門下一步的工作方向，從而確保產品運行的底層平臺能夠良好的支持最新技術。開發人員也會帶來相關的技術、知識和工作，幫助你們改善產品的運行環境，使其更加易于維護、簡潔有效。

有一些開發領域的概念，例如：“要根據API而非封閉的interface來構建工具”，分布式版本控制，驅動測試開發，以及諸如敏捷開發、看板管理(Kanban) 和Scrum等方法論。如果把這些概念應用在運維領域，同樣會產生革命性的變革。

Don’t be afraid of ideas and approaches from outside your domain – we can all learn things, even if it’s “let’s never do it that way again…!”, from how others do things and ultimately? Guess what? Yep, we’re all on the SAME team.

不要懼怕新點子和新技術。我們可以隨時隨地的向他人學習，哪怕是一句“我們再也不要那樣做了！” 也會讓我們從中獲益。盡管處于不同的部門，但是我們要共同學習、共同成長，這樣才能協同工作的更好！

Remember that interactions between people rank, in decreasing order of effectiveness (in IMHO but backed by some research):

按照從高到低的順序，有效的溝通方式應該是：

   1. Face to face  

1. 面對面交流 
 

   2. Video conference

2. 視頻會議 
 

   3. Phone

3. 電話 
 

   4. IM & IRC

4. 即時通訊軟件 
 

   5. Email

5. Email.
  

Process

工作中的流程

Don’t underestimate the power of process and automation.  Many shops do process engineering – ranging from hand-written lists to ISO9001. Those processes generally have one key flaw: they focus on the outcome and its inevitability.  A simple process might provision a host – Step 1 install machine, Step 2 cable machine, Step 3 install OS, etc, etc. Assuming all goes to process then at the end of Step x you will have a fully provisioned host. But what happens if it doesn’t go right?  If your process breaks or you receive some anomalous output how does your process deal with it?  

Instead think about process as a journey and map out the potential pitfalls and obstacles.  Treat your processes like applications and build error handling into them.  You can’t predict every application or operational pitfall or issue but you can ensure that if you hit one your process isn’t derailed.

不要低估流程和自動化的作用。很多公司都有自己的流程管理(process engineering)—— 從原始的筆錄到 ISO9001。但它們都存在一個關鍵的缺陷：過于理想化，它要求每個步驟都必須成功執行。例如：為了搭建一臺新主機，會有下列一套簡單的流程：步驟一：裝機（把各個硬件組裝到一起）。步驟二：接線、通電。步驟三：安裝操作系統。接下來還有步驟四、五、六。如果一切順利的話，第N步結束之后就會有一個功能完整、運行正常的新主機。但萬一有個流程沒跑通怎么辦？比如說在某個步驟斷了，走不下去了，或者在這一步得到了異常的輸出，有沒有另外的步驟來處理這個異常？

所以，流程絕對不會從頭到尾一帆風順，所以我們要把每一步流程都認真對待，找出所有潛在的問題和障礙。跟軟件產品一樣，在流程的管理中也要有異常處理。我們不必做到精確預見每一個問題，但一定要保證：即使流程出錯，它還能往下走。

Link process together across domains – software deployment, monitoring, capacity planning and other “operational” processes have their start in the development world.  Software deployment is the logical conclusion of the software development life cycle and should be viewed as such rather than a separate operational process. Another example is metrics and monitoring, it is hard to measure anything  without understanding the baselines and assumptions made in the development domain.  Joint processes also mean more opportunity for development and operations interaction, understanding and joint accountability. Finally, joint process development means single repositories for documentation and other opportunities for economies of scale.

把不同領域的所有流程串到一起。這些領域包括：部署、監控、能力計劃(capacity planning) 等等。從邏輯上講，“部署”是軟件開發周期的最后一環，所以它應該屬于“開發流程”，而非“運維流程”。另一個例子是度量和監控。在開發領域，如果不理解底線標準和估算，就什么評估都做不了。把開發部門和運維部門的流程銜接在一起，也會讓兩個部門更好的配合、相互理解、承擔共同的責任。最后還有個優點：文檔只需要一份而不是兩份（開發一份、運維一份），從而節省了資金。  

Automate, automate, automate. Build or make use of simple and extensible tools (make sure they have APIs and machine readable input and output – see James White’s Infrastructure Manifesto).  Use tools like Puppet (or others) to manage your configuration.  Remember to extend your automation umbrella cross-domain and end-to-end in your environment – manage development, testing, staging and production environments with the same tools and processes.  Not only does this have economies of scale benefits in support and management but it means you can test deployment and management alongside functionality as your application and new codes rolls toward production.

自動化，自動化，還是自動化。構建或使用簡單、可擴展的工具（確保提供API, 機器可讀的輸入、輸出 -- 參考 James White的文章：Infrastructure Manifesto）。使用Puppet一類的工具做配置管理。要擴展這些自動化工具，使其能夠支持多個領域（開發領域和運維領域），并且在產品的不同環境（開發環境、測試環境、發布環境和生產環境）中使用相同的工具(也叫end-to-end)。這樣不但會在產品支持和管理方面帶來經濟效益，而且也可以在編寫新代碼的同時，進行產品的發布和管理。

Finally, when building process and automation always keep the KISS principle in mind. Complexity breeds opportunities for error. Build simple processes and tools that are easy to implement, manage and maintain.

最后，在構建流程和自動化時，要把KISS原則牢記于心。越復雜就越易錯。只有簡單的流程和工具才易于實現、易于管理和易于維護。

Continuous Improvement

持續改進

Don’t stop innovating and learning.  Technology moves fast.  So do customer requirements. Build continuous improvement and integration into your tools and processes.  Here is a good place operations people can learn from (good) developers about practises like test-driven development.  A good example here is to build tests for your software deployment process and infrastructure.  They are often an application in their own right and should be developed and maintained correctly. Your monitoring could also be extended with behavioural testing to deliver better business value.  Look at using development domain tools, like Hudson for example, to explore and measure the operational domain.

不要停止創新和學習。當今技術發展的很快，客戶的需求也往往如此。把“持續改進和持續集成” 加入到你的工具和流程中去，這也是運維人員向（優秀的）開發人員學習的好途徑，可以學到諸如測試驅動開發等最佳實踐。例如：可以向你的部署流程中加入單元測試。做監控時也應該增加些行為測試，提高交付質量。嘗試用開發領域中的工具（例如Hudson）在運維領域中做些工作（例如瀏覽數據(explore)、測量性能(measure)等等）。

Learn from mistakes and from outages.  Seek root cause aggressively AND cross-domain.  If you have an outage and a post-incident review then bring development and operational teams together to review the incident.  Sometimes some simple code refactoring can save making infrastructure changes.  Work together to fix root cause, treat it with the same process you develop to conduct project to production software deployment, rather than relegating them to incident review reports or batting issues between teams.

要不斷的總結教訓。要積極主動的、在不同領域尋找錯誤的根源。 一旦收到錯誤報告，就果斷把開發小組和運維小組找來，一起解決這個問題。有時候開發人員很簡單的幾次代碼重構，就可以很好的避免底層運行環境的改變，減少運維人員的負擔。總之，遇到問題時，開發部門和運維部門要密切配合、共同解決，而不是互相推諉、踢皮球。

Me

對我來說...

Finally, for me DevOps is about people and nature of the environment you want to work in.  The best thing about the movement for me is that it is trying to foster behaviours and environments where people work together towards joint goals rather than at cross-purposes or at odds.  That’s a world I’d much rather use my skills in.

最后，對我來說，DevOps 的主要內容是：跟誰共同工作、如何共同工作。它最吸引我的地方就是致力于把不同部門不同分工的人召集到一起，共同努力解決問題。這樣的工作環境，是我所憧憬的樂園。