6月 30 2009

NEC Express5800/110Ge と VMWare ESXi で仮想サーバを作ってみる(2) NetBSDミラーリング編

Published by HoLY at 23:59:18 under tech

しばらく間が開きましたが NEC Express5800/110Ge と VMWare ESXi で仮想サーバを作ってみる(1) の続き。

今回構築したシステムは iSCSI のターゲットを NetBSD 上に構築、そのストレージは RAID1 で、という要件。その2つの設定だけだったので、 pkgsrc とか一切なし。容量も小さいので、HDD 全部外して USB メモリ上にインストールすることにする。

その構築過程のメモ。

インストール

インストールは ISO イメージを 国内のミラー から持ってきてCDに焼いてブートして、 USB メモリ挿した 110Ge にインストールするだけで、特に変わったところは無し。 普通の HDD 付けてる場合はどちらにインストールするか慣れないと間違えるかも。(USB メモリは SCSI 扱いなので sd? 、IDE な HDD は wd? となる)

パーティションは / のみまるっと切って終わり。 swap も無し。(単なるファイルサーバだからね)

ミラーリング(1)

RAID は raidframe という機構を使う。まず raid1 組む。都合により最初から HDD を2発用意できなかったので、まず片肺運転ということにする設定をする。

コマンドについてはman 8 raidctlを参照。

まず使用する HDD を fdisk でフォーマット。

# fdisk -u0a /dev/rwd0
fdisk: primary partition table invalid, no magic in sector 0
Disk: /dev/rwd0d
NetBSD disklabel disk geometry:
cylinders: 1240341, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
total sectors: 1250263728

BIOS disk geometry:
cylinders: 1023, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
total sectors: 1250263728

Do you want to change our idea of what BIOS thinks? [n] 

Partition 0:
<UNUSED>
The data for partition 0 is:
<UNUSED>
sysid: [0..255 default: 169] 
start: [0..77825cyl default: 63, 0cyl, 0MB] 
size: [0..77825cyl default: 1250263665, 77825cyl, 610480MB] 
bootmenu: [] 
Do you want to change the active partition? [n] y
Choosing 4 will make no partition active.
active partition: [0..4 default: 0] 
Are you happy with this choice? [n] y

We haven't written the MBR back to disk yet.  This is your last chance.
Partition table:
0: NetBSD (sysid 169)
    start 63, size 1250263665 (610480 MB, Cyls 0-77825/80/63), Active
        PBR is not bootable: All bytes are identical (0x00)
1: <UNUSED>
2: <UNUSED>
3: <UNUSED>
Bootselector disabled.
First active partition: 0

パーティション全体を使うので、disklabel は確認だけして操作せず。

次、RAID の設定。 /etc/raid0.conf を以下の内容で作成。

START array
1 2 0

START disks
/dev/wd0e
absent

START layout
128 1 1 1

START queue
fifo 100

START disks のところの2行目が absent になっている。これによって必要な台数(2、START ARRAY で指定) に足りない部分を「とりあえず無しで」という状態にする。

この内容をカーネルに知らせる。 ID は適当…。

# raidctl -C /etc/raid0.conf raid0
# raidctl -I 20090620 raid0
# raidctl -iv raid0

そうすると片肺運転で raid0 デバイスが使えるようになるはず。

# raidctl -s raid0 
Components:
           /dev/wd0e: optimal
          component1: failed
No spares.
Component label for /dev/wd0e:
   Row: 0, Column: 0, Num Rows: 1, Num Columns: 2
   Version: 2, Serial Number: 2009052301, Mod Counter: 195
   Clean: No, Status: 0
   sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 1228795648
   RAID Level: 1
   Autoconfig: No
   Root partition: No
   Last configured as: raid0
component1 status is: failed.  Skipping label.
Parity status: DIRTY
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.

disklabel を確認、設定。こんな感じになった。

# /dev/rraid0d:
type: RAID
disk: raid
label: fictitious
flags:
bytes/sector: 512
sectors/track: 128
tracks/cylinder: 4
sectors/cylinder: 512
cylinders: 0
total sectors: 1250263728
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0 

4 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 a: 536870911         0     4.2BSD      0     0     0  # (Cyl.      0 - 1048575*)
 d: 536870911         0     unused      0     0        # (Cyl.      0 - 1048575*)

newfs する。

# newfs /dev/raid0a

/etc/fstab を設定。 /storage というディレクトリを作成してそこにマウントしてる。

/dev/raid0a     /storage        ffs     rw      1 1

# mount /dev/raid0a /storage

これでとりあえず運用可能。

ミラーリング(2)

2台目の HDD をゲットしたので組み込む。異常時の交換手順の確認も兼ねてる。

wd1 で認識。wd0 と同様に wd1 も fdisk 。

raid0 に wd1e を追加。スペアとして認識される。

# raidctl -v -a /dev/wd1e raid0
# raidctl -s raid0
Components:
           /dev/wd0e: optimal
          component1: failed
Spares:
           /dev/wd1e: spare
Component label for /dev/wd0e:
   Row: 0, Column: 0, Num Rows: 1, Num Columns: 2
   Version: 2, Serial Number: 2009052301, Mod Counter: 195
   Clean: No, Status: 0
   sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 1228795648
   RAID Level: 1
   Autoconfig: No
   Root partition: No
   Last configured as: raid0
component1 status is: failed.  Skipping label.
/dev/wd1e status is: spare.  Skipping label.
Parity status: DIRTY
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.

component1 を Rebuild する。これで spare のディスクにコピーが始まる。

# raidctl -F component1 raid0
# raidctl -s raid0
# raidctl -s raid0
Components:
           /dev/wd0e: optimal
          component1: reconstructing
Spares:
           /dev/wd1e: used_spare
Component label for /dev/wd0e:
   Row: 0, Column: 0, Num Rows: 1, Num Columns: 2
   Version: 2, Serial Number: 2009052301, Mod Counter: 196
   Clean: No, Status: 0
   sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 1228795648
   RAID Level: 1
   Autoconfig: No
   Root partition: No
   Last configured as: raid0
component1 status is: reconstructing.  Skipping label.
Component label for /dev/wd1e:
   Row: 0, Column: 0, Num Rows: 0, Num Columns: 0
   Version: 0, Serial Number: 0, Mod Counter: 0
   Clean: No, Status: 0
   sectPerSU: 0, SUsPerPU: 0, SUsPerRU: 0
   Queue size: 0, blocksize: 0, numBlocks: 0
   RAID Level: ^@
   Autoconfig: No
   Root partition: No
   Last configured as: raid0
Parity status: DIRTY
Reconstruction is 0% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.

コピー中は、raidctl -S で過程を見ることができる。

# raidctl -S raid0
Reconstruction is 1% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.
Reconstruction status:

  1% |                                       | ETA: 01:35:27 /

変更する必要は無いかもしれないけれども気持ちの問題で /etc/raid0.conf を書き換え。absent -> /dev/wd1e に。

終わったらリブート。

# raidctl -s raid0
Components:
           /dev/wd0e: optimal
           /dev/wd1e: optimal
No spares.
Component label for /dev/wd0e:
   Row: 0, Column: 0, Num Rows: 1, Num Columns: 2
   Version: 2, Serial Number: 2009052301, Mod Counter: 204
   Clean: No, Status: 0
   sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 1228795648
   RAID Level: 1
   Autoconfig: No
   Root partition: No
   Last configured as: raid0
Component label for /dev/wd1e:
   Row: 0, Column: 1, Num Rows: 1, Num Columns: 2
   Version: 2, Serial Number: 2009052301, Mod Counter: 204
   Clean: No, Status: 0
   sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
   Queue size: 100, blocksize: 512, numBlocks: 1228795648
   RAID Level: 1
   Autoconfig: No
   Root partition: No
   Last configured as: raid0
Parity status: clean
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.

# dmesg
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 5.0 (GENERIC) #0: Sun Apr 26 18:50:08 UTC 2009
        builds@b6.netbsd.org:/home/builds/ab/netbsd-5-0-RELEASE/i386/200904260229Z-obj/home/builds/ab/netbsd-5-0-RELEASE/src/sys/arch/i386/compile/GENERIC
total memory = 1021 MB
avail memory = 992 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
NEC           Express5800/110Ge [N8100-1447Y]                        (FR1.3      )
mainbus0 (root)
cpu0 at mainbus0 apid 0: Intel 686-class, 1795MHz, id 0x10661
ioapic0 at mainbus0 apid 1: pa 0xfec00000, version 20, 24 pins
acpi0 at mainbus0: Intel ACPICA 20080321
acpi0: X/RSDT: OemId <PTLTD ,    XSDT  ,06040000>, AslId < LTP,00000000>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
ACPI-Fast 24-bit timer
npx1 at acpi0 (MATH, PNP0C04)
npx1: io 0xf0-0xfe irq 13
npx1: reported by CPUID; using exception 16
hpet0 at acpi0 (HPET, PNP0103)
hpet0: mem 0xfed00000-0xfed003ff irq 0,8
timecounter: Timecounter "hpet0" frequency 14318179 Hz quality 2000
pcppi1 at acpi0 (SPKR, PNP0800)
pcppi1: io 0x61
midi0 at pcppi1: PC speaker (CPU-intensive output)
sysbeep0 at pcppi1
attimer1 at acpi0 (TIMR, PNP0100): AT Timer
attimer1: io 0x40-0x43,0x50-0x53
pckbc1 at acpi0 (KBC0, PNP0303): kbd port
pckbc1: io 0x60,0x64 irq 1
pckbc2 at acpi0 (MSE0, PNP0F13): aux port
pckbc2: irq 12
FWH (INT0800) at acpi0 not configured
COMA (PNP0501) at acpi0 not configured
COMB (PNP0501) at acpi0 not configured
LPT (PNP0401) at acpi0 not configured
acpibut0 at acpi0 (PWRB, PNP0C0C): ACPI Power Button
apm0 at acpi0: Power Management spec V1.2
attimer1: attached to pcppi1
pckbd0 at pckbc1 (kbd slot)
pckbc1: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: vendor 0x8086 product 0x29f0 (rev. 0x01)
ppb0 at pci0 dev 1 function 0: vendor 0x8086 product 0x29f1 (rev. 0x01)
ppb0: unsupported PCI Express version
pci1 at ppb0 bus 1
pci1: no spaces enabled!
wm0 at pci0 dev 25 function 0: 82801I (AMT) LAN Controller, rev. 2
wm0: interrupting at ioapic0 pin 16
wm0: PCI-Express bus
wm0: FLASH
wm0: Ethernet address 00:24:1d:3f:36:02
igphy0 at wm0 phy 1: i82566 10/100/1000 media interface, rev. 0
igphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
uhci0 at pci0 dev 26 function 0: vendor 0x8086 product 0x2937 (rev. 0x02)
uhci0: interrupting at ioapic0 pin 20
usb0 at uhci0: USB revision 1.0
uhci1 at pci0 dev 26 function 1: vendor 0x8086 product 0x2938 (rev. 0x02)
uhci1: interrupting at ioapic0 pin 21
usb1 at uhci1: USB revision 1.0
uhci2 at pci0 dev 26 function 2: vendor 0x8086 product 0x2939 (rev. 0x02)
uhci2: interrupting at ioapic0 pin 22
usb2 at uhci2: USB revision 1.0
ehci0 at pci0 dev 26 function 7: vendor 0x8086 product 0x293c (rev. 0x02)
ehci0: interrupting at ioapic0 pin 22
ehci0: EHCI version 1.0
ehci0: companion controllers, 2 ports each: uhci0 uhci1 uhci2
usb3 at ehci0: USB revision 2.0
ppb1 at pci0 dev 28 function 0: vendor 0x8086 product 0x2940 (rev. 0x02)
pci2 at ppb1 bus 5
pci2: no spaces enabled!
ppb2 at pci0 dev 28 function 4: vendor 0x8086 product 0x2948 (rev. 0x02)
pci3 at ppb2 bus 13
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
jmide0 at pci3 dev 0 function 0: vendor 0x197b product 0x2368
jmide0: 1 PATA port
jmide0: interrupting at ioapic0 pin 16
jmide0: PCI IDE interface used
jmide0: bus-master DMA support present
jmide0: primary channel wired to native-PCI mode
jmide0: primary channel is PATA
atabus0 at jmide0 channel 0
jmide0: secondary channel wired to native-PCI mode
jmide0: secondary channel is unused
ppb3 at pci0 dev 28 function 5: vendor 0x8086 product 0x294a (rev. 0x02)
pci4 at ppb3 bus 15
pci4: no spaces enabled!
uhci3 at pci0 dev 29 function 0: vendor 0x8086 product 0x2934 (rev. 0x02)
uhci3: interrupting at ioapic0 pin 19
usb4 at uhci3: USB revision 1.0
uhci4 at pci0 dev 29 function 1: vendor 0x8086 product 0x2935 (rev. 0x02)
uhci4: interrupting at ioapic0 pin 20
usb5 at uhci4: USB revision 1.0
uhci5 at pci0 dev 29 function 2: vendor 0x8086 product 0x2936 (rev. 0x02)
uhci5: interrupting at ioapic0 pin 21
usb6 at uhci5: USB revision 1.0
ehci1 at pci0 dev 29 function 7: vendor 0x8086 product 0x293a (rev. 0x02)
ehci1: interrupting at ioapic0 pin 19
ehci1: EHCI version 1.0
ehci1: companion controllers, 2 ports each: uhci3 uhci4 uhci5
usb7 at ehci1: USB revision 2.0
ppb4 at pci0 dev 30 function 0: vendor 0x8086 product 0x244e (rev. 0x92)
pci5 at ppb4 bus 17
pci5: i/o space, memory space enabled
vga1 at pci5 dev 3 function 0: vendor 0x18ca product 0x0020 (rev. 0x00)
wsdisplay0 at vga1 kbdmux 1: console (80x25, vt100 emulation), using wskbd0
wsmux1: connecting to wsdisplay0
drm at vga1 not configured
ichlpcib0 at pci0 dev 31 function 0
ichlpcib0: vendor 0x8086 product 0x2916 (rev. 0x02)
timecounter: Timecounter "ichlpcib0" frequency 3579545 Hz quality 1000
ichlpcib0: 24-bit timer
ichlpcib0: TCO (watchdog) timer configured.
ahcisata0 at pci0 dev 31 function 2: vendor 0x8086 product 0x2922
ahcisata0: interrupting at ioapic0 pin 17
ahcisata0: AHCI revision 1.2, 6 ports, 32 command slots, features 0xe322e0c0
atabus1 at ahcisata0 channel 0
atabus2 at ahcisata0 channel 1
atabus3 at ahcisata0 channel 2
atabus4 at ahcisata0 channel 3
atabus5 at ahcisata0 channel 4
atabus6 at ahcisata0 channel 5
ichsmb0 at pci0 dev 31 function 3: vendor 0x8086 product 0x2930 (rev. 0x02)
ichsmb0: interrupting at ioapic0 pin 17
iic0 at ichsmb0: I2C bus
isa0 at ichlpcib0
lpt0 at isa0 port 0x378-0x37b irq 7
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
uhub0 at usb0: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhub1 at usb1: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhub2 at usb2: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhub3 at usb3: vendor 0x8086 EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub3: 6 ports with 6 removable, self powered
uhub4 at usb4: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub4: 2 ports with 2 removable, self powered
uhub5 at usb5: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub5: 2 ports with 2 removable, self powered
uhub6 at usb6: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub6: 2 ports with 2 removable, self powered
uhub7 at usb7: vendor 0x8086 EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub7: 6 ports with 6 removable, self powered
atapibus0 at atabus0: 2 targets
cd0 at atapibus0 drive 0: <MATSHITADVD-RAM SW-9590A, , X100> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 4 (Ultra/66)
cd0(jmide0:0:0): using PIO mode 4, Ultra-DMA mode 4 (Ultra/66) (using DMA)
ahcisata0 port 0: device present, speed: 3.0Gb/s
ahcisata0 port 1: device present, speed: 3.0Gb/s
wd0 at atabus1 drive 0: <SAMSUNG HD642JJ>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 596 GB, 1240341 cyl, 16 head, 63 sec, 512 bytes/sect x 1250263728 sectors
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 7
wd0(ahcisata0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
wd1 at atabus2 drive 0: <WDC WD6400AAKS-22A7B2>
wd1: drive supports 16-sector PIO transfers, LBA48 addressing
wd1: 596 GB, 1240341 cyl, 16 head, 63 sec, 512 bytes/sect x 1250263728 sectors
wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd1(ahcisata0:1:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
umass0 at uhub7 port 4 configuration 1 interface 0
umass0: Kingston DT 101 II, rev 2.00/2.00, addr 2
umass0: using SCSI over Bulk-Only
scsibus0 at umass0: 2 targets, 1 lun per target
sd0 at scsibus0 target 0 lun 0: <Kingston, DT 101 II, 1.00> disk removable
sd0: 7664 MB, 3936 cyl, 16 head, 32 sec, 512 bytes/sect x 15695872 sectors
Kernelized RAIDframe activated
pad0: outputs: 44100Hz, 16-bit, stereo
audio0 at pad0: half duplex
boot device: sd0
root on sd0a dumps on sd0b
root file system type: ffs
uhidev0 at uhub5 port 1 configuration 1 interface 0
uhidev0: vendor 0x0566 product 0x3002, rev 1.10/1.00, addr 2, iclass 3/1
ukbd0 at uhidev0
raid0: Component /dev/wd0e being configured at col: 0
         Column: 0 Num Columns: 2
         Version: 2 Serial Number: 2009052301 Mod Counter: 199
         Clean: Yes Status: 0
raid0: Component /dev/wd1e being configured at col: 1
         Column: 1 Num Columns: 2
         Version: 2 Serial Number: 2009052301 Mod Counter: 199
         Clean: Yes Status: 0
WARNING: truncating disk at c 1 to 1228795658 blocks
raid0: RAID Level 1
raid0: Components: /dev/wd0e /dev/wd1e
raid0: Total Sectors: 1228795648 (599997 MB)
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub5 port 1 configuration 1 interface 1
uhidev1: vendor 0x0566 product 0x3002, rev 1.10/1.00, addr 2, iclass 3/0
uhidev1: 4 report ids
uhid0 at uhidev1 reportid 2: input=1, output=0, feature=0
uhid1 at uhidev1 reportid 3: input=3, output=0, feature=0
uhid2 at uhidev1 reportid 4: input=1, output=0, feature=0
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)

長くなったので iSCSI については次に。

参考

No responses yet

Trackback URI | Comments RSS

Leave a Reply