Calling all I.T. Pro's and super geeks, help needed

[Kayos]

The following post is very very geeky so I apologise in advance, I don't use many forums and really don't want to sign up to a computer forum

Im currently rebuilding my server, using some of the leftovers from my PC upgrade (tight Yorkshireman alert) and im at the final details stage that I know could make a huge difference

First question is about IP addresses, my "normal" network uses 192.168.0.whatever, using my router as the DHCP server, as ive ditched Samba (yay!!!!) and gone to iSCSI for my main PC ive also acquired a pair of Broadcom dual port NIC's to connect the server direct to the PC for super fast connections, however im not sure what IP's to give the new interfaces.
They will all be static so can I just go 192.168.1.1to4? or is it not that simple? ive seen a suggestion of 10.10.whatever.1-4 but I don't know enough to know what's best
The other NIC will be on the normal network so I can admin the server from anywhere and not just the main desktop

The second question is even more geeky! Ive decided to run ZFS to stop the so called bit rot, as most of my files are hardly touched, I just don't want to loose them. I have 4 x 3TB drives installed and I don't know what raid setup to use
My initial thought was the ZFS equivalent of RAID10, but what about RAIDZ? supposed to be like RAID5 but faster as ZFS will read from the mirrored drives in stripes, however reading a checksum, and presumably rebuilding, is limited to the speed of a single drive
im happy with 6TB of storage, more is better of course, I just want it safe from a failure and not so slow I don't bother doing backups, the drives are enterprise grade, pulled from new cancelled order servers so shouldn't fail but I have an identical spare just in case and the whole point of the server is to keep my stuff safe.

Not using ZFS or iSCSI isn't an option, Samba makes me want to hurt myself and ZFS is better than hard/soft/fakeRAID in this setup.

Sorry for the geekfest, im firing the plasma and the mig up tomorrow to make up for it

 

[brianblock]

As far as the IP goes just avoid the default IP range of the dhcp and all will be fine
I usually start assigning static IP at 192.168.1.100

 

[zx9]

The following post is very very geeky so I apologise in advance, I don't use many forums and really don't want to sign up to a computer forum

Im currently rebuilding my server, using some of the leftovers from my PC upgrade (tight Yorkshireman alert) and im at the final details stage that I know could make a huge difference

First question is about IP addresses, my "normal" network uses 192.168.0.whatever, using my router as the DHCP server, as ive ditched Samba (yay!!!!) and gone to iSCSI for my main PC ive also acquired a pair of Broadcom dual port NIC's to connect the server direct to the PC for super fast connections, however im not sure what IP's to give the new interfaces.
They will all be static so can I just go 192.168.1.1to4? or is it not that simple? ive seen a suggestion of 10.10.whatever.1-4 but I don't know enough to know what's best
The other NIC will be on the normal network so I can admin the server from anywhere and not just the main desktop

The second question is even more geeky! Ive decided to run ZFS to stop the so called bit rot, as most of my files are hardly touched, I just don't want to loose them. I have 4 x 3TB drives installed and I don't know what raid setup to use
My initial thought was the ZFS equivalent of RAID10, but what about RAIDZ? supposed to be like RAID5 but faster as ZFS will read from the mirrored drives in stripes, however reading a checksum, and presumably rebuilding, is limited to the speed of a single drive
im happy with 6TB of storage, more is better of course, I just want it safe from a failure and not so slow I don't bother doing backups, the drives are enterprise grade, pulled from new cancelled order servers so shouldn't fail but I have an identical spare just in case and the whole point of the server is to keep my stuff safe.

Not using ZFS or iSCSI isn't an option, Samba makes me want to hurt myself and ZFS is better than hard/soft/fakeRAID in this setup.

Sorry for the geekfest, im firing the plasma and the mig up tomorrow to make up for it

Forget all that lot just buy acid free negative file pages.

 

[Kayos]

Forget all that lot just buy acid free negative file pages.

Done that too

Problem is I cant wet print colour, each scanned 35mm shot is around 150mb each, x 36 x 3 to 5 a week...........

You can see why my 2tb server had to go

 

[rikrobson]

If you are connecting to pc and server directly together you need to put that connection on a different subnet. There will be no gateway either on the subnet. Set up static ips on the two nics the mask won't matter so use a /24
You wanted use zfs1 which will use on redundant disk like raid 5

I'd recommend using greens on the server as it will make the configuration easier?. It can be installed on a pen drive giving you all disks for storage

 

[Kayos]

If you are connecting to pc and server directly together you need to put that connection on a different subnet. There will be no gateway either on the subnet. Set up static ips on the two nics the mask won't matter so use a /24
You wanted use zfs1 which will use on redundant disk like raid 5

I'd recommend using greens on the server as it will make the configuration easier?. It can be installed on a pen drive giving you all disks for storage

That's what I was thinking with the IP's, Its all connected temporary now but wanted a full run through before the server is put back in its cupboard

I shall configure some disks with ZFS1 in the morning and do some speed/stress test Thanks

I'm running a version of NAS4FREE but ive modified it hugely over the last 2 years, I like BSD, its like Linux but it rarely gets in my way

 

[Cyberprog]

Personally my setup uses a pfSense router and all my networks sit in the 10.0.x.x range. But I'm a little puzzled why you're using iSCSI and yet don't really understand the underlying networking. It's like running before you can walk.

If you're trying to protect your data, then RAID1 would be the way to go. I very much doubt you need RAID1+0, and if you're trying for 6TB of space, then that's a lot of storage to make "fast".

I use Drobo NAS devices for my storage, just throw disks at it when needed, and pop them out and upgrade when I need more space. It just handles it all.

On my desktop, I run a SSD for the OS and a RAID1 2TB set for data. That suffices. Most of our servers run RAID1, a few run RAID5, and a few run that with SAS 10K or 15K drives.

Think about what you want to achieve, and configure things to suit. SATA disk is cheap, and great at holding bulky data for long periods, while SAS is better for stuff that needs a bit more performance, while super quick database access/OS boot is what SSD's are good at, though you keep those setup so you're mirroring data between machines usually in case of loss of an SSD. They also don't cope terribly well at the cheap end of the market with RAID.

 

[Kayos]

I do understand the networking, I just didn't think it was as simple as I first thought, turns out it is

iSCSI is fast and easy to set up, the server only serves the desktop and Samba has a habit of making me hurt things so it's the way to go, plus having it on its own network avoids slow downs due to rest of the network

The server runs in embedded mode, so it boots from a usb stick and runs completely in ram and has the 4 3TB drives connected to the motherboard, using ZFS allows me to dynamically change the raid modes, but having only ever used (fake)raid 0 and 1 previously I wanted more info on what would be better

The desktop runs 2 x 250gb Raid0 for boot and a working partition (scans and memory card imports) then my current files are on a 2TB raid inside the desktop, this is backed up to the server after every shoot, effectively giving me 4 copies of each shot
The old server has 2TB raid1 (still in use until I'm happy with the new one) but using Samba and only one NIC it can drop to 25mb/s and considering today I shot 3 films (16gb ish) and 2 16gb memory cards I need more than that!

 

[skotl]

As above re the IPs. Having two NICs on the same subnet will lead to all sorts of routing problems (unless you are teaming the NICs so that they share the same IP address).
If your router / switch supports VLANs then make sure that the iscsi network is on its own VLAN so that traffic from the "internet" VLAN doesn't bleed into the iscsi VLAN.

Re RAID, to be honest I would just mirror them and use the 3rd disk for something else.

 

[Kayos]

Just to clarify, the server connects to the desktop with 2 crossover cables as both server and desktop have matching iSCSI aware Broadcom twin port NICs

That's the 192.168.1.1-4 network (server is .1 and .2, desktop is .3 and .4)

They also both connect to the "normal" network with the onboard nics through the router

This is the 192.168.0.whatever the dhcp server says (server's ip is reserved in the router)

My main question was is that sufficient to stop issues with the network? It's all working fine sat on my desk but things have a habit of not working once they are installed under the stairs and not easy to get at

 

[Cyberprog]

As long as each interface has a unique address, that's fine, unless as skotl says you're teaming them.

 

[brewdexta]

as long as they have /24 masks then you should be fine, the "normal" and the storage network will be on different broadcast domains

 

[skotl]

Ah wait....

At first I thought you were talking about a NAS device talking iscsi to a server, but now I'm not clear. Is all of the storage direct-attached (i.e. not via iscsi) or are you using the LAN to connect the server's storage via iscsi?

Generally, you should never connect two NICs on a single device to the same subnet as you'll get bridging / routing issues.

If the server is talking to a NAS then put them on their own subnet, and have the client devices talk to the server and internet router on a separate subnet.

 

[brewdexta]

a diagram would be good

 

[Kayos]

Yes, the server is essentially an iSCSI device the desktop connects to so I can backup my stuff quickly and reliably

The NICs are designed to work like this, they are TOE (tcip offload engine) and using both ports makes them faster, I'm getting sustained 160mb/s with folders containing a few hundred 25mb files, slightly less on the 150mb files.

No diagrams, for the minute imagine a plate of spaghetti with servers and PCs attached for now

 

[brewdexta]

ah, kernel bypass cards, Mellanox, Solar Flare, Chelsio?

 

[skotl]

Yes, the server is essentially an iSCSI device the desktop connects to so I can backup my stuff quickly and reliably

The NICs are designed to work like this, they are TOE (tcip offload engine) and using both ports makes them faster, I'm getting sustained 160mb/s with folders containing a few hundred 25mb files, slightly less on the 150mb files.

No diagrams, for the minute imagine a plate of spaghetti with servers and PCs attached for now

So... it's a NAS, rather than a server (as a Windows, NFS or Samba device would be)? Where are the disks? Are they connected directly to the "server" and is that server a PC in disguise?
I'm still a little confused - iscsi connects peripherals so I'm seeing it as a stretch if the server is effectively a disk array that is mounted by the desktop.

However, that doesn't negate the earlier advice; storage, such as iscsi connections, should be on a dedicated LAN which means a dedicated subnet. (and you shouldn't have two NICs on the same subnet, on a single machine).

 

[Kayos]

I keep calling it a server as the old one was (is) the new one is more NAS and it's kind of a server/pc hybrid, it's a server motherboard that I've shoehorned into an ATX case, and used the cpu and ram from my old pc (ecc would be better but I had 8gb of ram on my desk, it wasn't getting wasted)

Another smaller array is used as a PS3 and Xbox media server over the normal network, the iSCSI is on its own network to the desktop

It's (loosly) based on nas4free but I have stripped everything I don't need out of it, it runs in about 200mb of ram super fast

 

[Kayos]

I've resolved a few issues I had put down to Windows 10 on the desktop, turns out my new SSD's were faulty. Now replaced its all up and running

ive stopped calling it a server as it now a dedicated NAS, the old one did DHCP, port forwarding as well a uPNP and a few other bits, I no longer need uPNP and everything else has moved to the new router.

The NAS and my desktop have matching dual port Broadcom nic's that are PCIe x4 plugged into x8 slots in both machines, after more research I went with a simple logical IP setup

Nas:
Management nic 192.168.0.100 (DHCP reservation on the router)
Port 0: 10.10.10.10
Port 1: 10.20.20.20

Desktop
Port 0: 10.10.10.11
Port 1: 10.20.20.21

The 2 are connected with 2 crossover cables and the NAS is now hiding in its cupboard, vented to outside so plenty of fresh air to keep it cool

The iSCSI was setup to just utilise the dual nics and with multipath enabled im now getting sustained 160mb/s WRITE speeds and 190+ reads in real world situations, with even more when running benchmarks!

That's with a 3TB mirror setup, may be faster with stripe but didn't feel it was worth it, I can add extra drives as my needs grow and hopefully prolong their life

 

[ukracer]

I'm still a little confused -.

Your confused.....have a thought for us mere mortals. lol