Amazon EC2/S3 Tutorial

主機自動備份-snapshot

以 EC2 API 方式對 Instance 定期作 snapshot，並只保持最新期間的備份檔。

作者網站：http://tiger-fish.com/blog/automated...ing-amazon-ec2
專案首頁：http://code.google.com/p/ec2-delete-old-snapshots/

主要的 EC2 API 有兩個：

1. 新增 snapshot：這個 API 有兩種，一個是標準的由 Java 開發的 EC2 API Tools，不過這工具比較肥大；另一個是由 PHP 開發的 AWS SDK - CreateSnapshotSample.php。
2. 刪除舊的 snapshot (ec2-manage-snapshot.php)

Cron Script:

Callstack:
    at User:Alang/IT_專案工作/Amazon_EC2//S3_Tutorial

MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException: reference to undefined name 'syntax' Exception of type 'MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException' was thrown.
  at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptVar expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0 
  at MindTouch.Deki.Script.Expr.DekiScriptVar.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0 
  at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Evaluate (MindTouch.Deki.Script.Expr.DekiScriptAccess expr, DekiScriptExpressionEvaluationState state, Boolean evaluateProperties) [0x00000] in <filename unknown>:0 
  at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptAccess expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0 
  at MindTouch.Deki.Script.Expr.DekiScriptAccess.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0 
  at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptCall expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0 
  at MindTouch.Deki.Script.Expr.DekiScriptCall.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0 
  at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptSequence expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0 
  at MindTouch.Deki.Script.Expr.DekiScriptSequence.VisitWith[DekiScriptExpressionEvaluationState,Range] (IDekiScriptExpressionVisitor`2 visitor, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0 
  at MindTouch.Deki.Script.Compiler.DekiScriptExpressionEvaluation.Visit (MindTouch.Deki.Script.Expr.DekiScriptReturnScope expr, DekiScriptExpressionEvaluationState state) [0x00000] in <filename unknown>:0 

CreateSnapshot.php

Firefox Extensions

• Elasticfox
• S3Fox

如何取得 AWS 的各種認證資訊

在 AWS 的服務會用到的認證種類區分三種：

1. Access Key            - 包括有 Access Key ID 及 Secret Access Key 兩個資訊。
2. X.509 Certificates  - 有關 AMI 的操作時會需要用到。
3. Key Pairs                - 用來連接 Instance 時之用
    

 Access Key ID 與 Secret Access Key 是 AWS 基本的身份認證，只要在網頁上完成註冊，就會有這兩組資訊。
要取得以上這些資訊，可以點選以下連結後，輸入 AWS 的帳號/密碼 即可。

http://aws-portal.amazon.com/gp/aws/...ion=access-key

註：key pairs 的管理不用透過這個連結，在 AWS Console 也能操作。

使用 Putty 登入 Instance

1. 在 AWS Console > Key Pairs，新增一組 key pair，並下載 xxx.pem 到本地電腦。(key pair 只有在新增時才能被下載，之後就無法再下載，如果遺失 key pair，就重新再新增一個，不過別忘了，也要一併修改 Instance  的設定)
2. 下載 PuTTYgen，這工具將用來產生 PuTTY Private Key file(*.ppk)
   執行 PuTTYgen > Conversions > Import key, 載入 xxx.pem > Save private key(提示沒有輸入 key passphrase, 按 yes) > 存檔成 xxx.ppk
3. 開啟 PuTTY > 輸入 IP > Connection type: SSH >
   Connection > SSH > Auth > Private key file for authentication: 載入 xxx.ppk > Open 連線
4. 輸入 root，不用密碼就能進入 console。

( http://docs.amazonwebservices.com/AW...tml?putty.html )

Creating AMI from existing instance

Today I will describe in short how to create an AMI instance of EC2 and than upload it to S3 server. All you need to have is running instance of EC2 server (I’m using Debian) with EC2
tools preinstalled.

First of all, make sure that you don’t have any unnecessary files on your server filesystem which you don’t need to bundle in an image as it will only slow the process down and make
your costs of storing backup images higher.
Once you are sure that your instance is fine and EC2 tools are in place you have to transfer your private key file and certificate file to the server. Lets put the files under /root
directory.
If all is set up correctly, first of all, we create and image by running:

ec2-bundle-vol -d /mnt -k /root/PrivateKey.pem --cert /root/Certificate.pem -u 012345678901

The number after the -u flag is your Amazon account ID which you can find on the AWS website once logged in.
Running above command will take a bit and will create a number of files under /mnt directory.
Once we have the files and manifest ready (all happens auto-magically) we can start transferring the image onto S3 account.

In order to transfer your image on S3 you would have to create a bucket there first, for this you can use famous firefox plugin (S3 Organizer) or do it using Affirma’s library (previous post).

ec2-upload-bundle -b yourbucketname00404042009 -m /mnt/image.manifest.xml -a your_s3_access_key -s your_s3_secret_access_key

If for some reason you encouter any problems during upload but your bundling went ok, you can retry the upload using –retry flag:

ec2-upload-bundle -b yourbucketname00404042009 -m /mnt/image.manifest.xml -a your_s3_access_key -s your_s3_secret_access_key --retry

註冊備份好的 AMI image

ec2-register -K pk-*.pem -C cert-*.pem your-bucket/image.manifest.xml

NOTES:

＊ pk-*.pem, cert-*.pem 這兩個是 x509 的 key，需要透過 AWS Console 來產生。

＊ 註冊後的 AMI，可以在 Elasticfox > Images > Filter: My AMIs 看到。

安裝 EC2 API Tools

// 下載 JRE 1.6 - jre-6u21-linux-i586.bin

http://www.oracle.com/technetwork/ja...ads/index.html

// 安裝 JRE 1.6

# chmod u+x jre-6u21-linux-i586.bin
# ./jre-6u21-linux-i586.bin
# mv jre1.6.0_21/ /opt

// 設定 JRE

# vi /etc/profile.d/java.sh

export JAVA_HOME=/opt/jre1.6.0_21
export PATH=$JAVA_HOME/bin:$PATH 

// 下載 ec2-api-tools

http://developer.amazonwebservices.c...externalID=351

// 安裝 ec2-api-tools

unzip  ec2-api-tools.zip
mv ec2-api-tools-1.3-53907/ ~/.ec2
vi ~/.bashrc

# for EC2 API Tools
export EC2_HOME=~/.ec2
export PATH=$PATH:$EC2_HOME/bin 

如何增加 EBS volume 空間

( http://blog.edoceo.com/2009/02/amazo...w-storage.html )

基本流程：umount > snapshot > create an new volume from snapshot > attach > FS check with e2fsck > resize > FS check again > mount

NOTES:

要增加一個線上系統的 EBS volume，系統必須要停機，只是參照本章節的程序，可以讓你盡可能的縮短 downtime。

// Step1

假設 EBS volume 是用於儲存資料庫之類，請務必關閉資料庫後，umount EBS volume 再繼續後續的作業；反之，若儲存的資料類型是屬於 off-line，且沒有鎖檔的顧慮，可以不需要 umount EBS volume。

// Step2

 透過 EC2 command 或 Elasticfox 對 EBS volume 新增一份 snapshot

// Step3

透過 EC2 command 或 Elasticfox 對剛新增的 snapshot 新增一個 EBS volume

// Step4

透過 EC2 command 或 Elasticfox 附加(attach)新的 EBS volume 到一個 instance，例如為 /dev/sdb。

TIPS:

為了縮短 downtime，這裡可以另外啟動一個 instance，或者也可以用 production instance。

// Step5

SSH 登入 instance，執行：

# e2fsck -f /dev/sdb
# resize2fs -p /dev/sdb
# e2fsck -f /dev/sdb
# tune2fs -l /dev/sdb

// Step6

掛載 /dev/sdb，並檢查資料是否完整。

心得總結：

1. 依照本教學，需要 downtime 的作業只有作 snapshot 及 mount/umount 時。