The ATM network will consist of the two Catalyst 8510 ATM switches (essentially Lightstream 1010s with the new name). These switches are connected with an Oc12 link, to avoid bottlenecking. Each switch will support 16 of the nodes via Oc3 connections.
The FastEthernet network will be provided by the two Catalyst 2924M-XL switches. The link between these two switches will be provided by 3 FastEthernet connections aggregated into a single 300Mbps link via EtherChannel. Similar to the Oc12 link between the two ATM switches, this EtherChannel will avoid bottlenecking of traffic going between the switches. Each Catalyst 2924M-XL switch will support 16 of the nodes via 100Mbps FastEthernet connections.
It's important to note that these two networks will be separate. There will be no direct conversion from Ethernet frames to ATM cells, for example, unless packets originate on the FastEthernet network and are addressed to an ATM interface. In other words, these two networks will be routed and not bridged.
That network-to-network connectivity will be provided by the 7204 router. The router will have an Oc3 connection to the ATM network, and a 100Mbps link to the FastEthernet network. It will also have a link to NUNet, initially at 10Mbps, moving to 100Mbps when the NUNet infrastructure has been upgraded to provide the greater bandwidth.
It will be important to be certain that IP forwarding is not enabled in any of the cluster nodes. If any of the nodes is configured to forward packets, routing loops and nondeterministic network topology will result.
Since they will be routed, the ATM and FastEthernet networks should be given separate IP subnet ranges. Since Northeastern does not currently have two free Class C-sized subnets free, my suggestion is to use private networks (as defined in RFC1597). For the puposes of this document, I will use network numbers from reserved network 10, though, any of the private networks would work equally well.
Considering the number of end-stations that will be attached to each network, Class C-sized subnets (254 hosts per subnet) should easily suffice. For example, the ATM network could be assigned addresses in the 10.1.1.X range (with a subnet of 255.255.255.0) and the FastEthernet network the range 10.1.2.X (again subnetted 255.255.255.0).
When assigning addresses in a new network, it is always wise to consider an addressing scheme. To avoid confusion, each end station should be given the same host-address on both networks. For example, "Node 1" could be assigned address 10.1.1.1 for its ATM interface, and 10.1.2.1 for its FastEthernet interface. "Node 2" would be 10.1.1.2 (ATM) and 10.1.2.2 (FastEthernet), etc. Special machines, such as the SMP server and perhaps the RAID servers, could either be numbered with the rest, or (if desired) be given addresses higher in the subnets to make them easier to remember, eg 10.1.1.75 and 10.1.2.75, etc.
The router, as a special piece of networking equipment, should be given IP addresses outside the range of normal nodes. For example, 10.1.1.100 and 10.1.2.100. Other special equipment or networking hardware should also be given "special" host addresses, to distinguish them from the cluster nodes. For example, the ATM switches can run LANE clients for in-band network management, and Ethernet ports for out-of-band connectivity; the ATM LANE client on the first switch could be 10.1.1.101 and the FastEthernet port 10.1.2.101. The FastEthernet switches do not have ATM interfaces, but will require IP addresses on the FastEthernet network for management.
With different subnet numbers for the two networks, it will be easy to cause packets to flow on the appropriate network. Since each node will be connected to both networks, configuring an application using addresses in the 10.1.1 subnet would cause all traffic to flow over the ATM network, and using 10.1.2 addresses would send traffic over the FastEthernet network. As mentioned earlier, it will be important to have IP forwarding *disabled* everywhere except on the router to ensure correct behavior.
[The above behaviour relies on the nodes' operating system and its choice of which interface to use for transmition. Dual-homed hosts not involved in IP forwarding should choose the interface directly connected to a given subnet when sending packets to that subnet. Eg, a host with interfaces 10.1.1.1 (ATM) and 10.1.2.1 (FastEthernet) should send a packet addressed to 10.1.1.2 out the ATM interface.]
Using a mix of 10.1.1 and 10.1.2 addresses in an application would result in a more complicated traffic flow, with some packets using the ATM network and some the FastEthernet.
With each machine connected to both networks, very little in-cluster network traffic will actually travel across the router, because each host can send traffic directly to each network. In fact, forcing packets from one cluster node to another to go across the router would be somewhat involved, and would require either disabling one of the interfaces on the sending host, and/or modifying the routing tables on the nodes involved.
The difficulty with using private network addresses is that these cannot be routed to the rest of the university. It will not be possible, for example, to telnet to 10.1.1.1 from ECE or CCS and reach the ATM interface of Node 1. Cisco's Network Address Translation capabilities on the router may be able to provide a solution, but at the very least it will be possible to have inbound telnet connections via a two-step process: telnet first to the outside address of the router, and from there telnet to the private address of a node.
This level of connectivity is admittedly minimal. Hopefully, a better solution can be found. If a range of official Northeastern IP addresses becomes available, that would be the cleanest and easiest solution.
The Catalyst 8510 ATM switches are configured with one Oc12 interface each. One switch has 16 Oc3 interfaces, while the other has 20. The switches have 3 and 2 slots free, respectively, to add more interfaces in the future if desired.
The Catalyst 2924M-XL switches have 24 non-modular ports of auto-sensing 10/100 Ethernet. In addition, they have 2 module slots which support additional FastEthernet or (in the future) GigabitEthernet or ATM connectivity. None of these module slots are in use currently.
The 7204 router has a 4-port 10BaseT card, a 2-port 100BaseTX card, and a 1-port Oc3 ATM card. The 7204 has one slot free.
The physical configuration of the network will be simple. MultiMode Fiber jumpers (with SC connections at both ends) will connect each node's ATM NIC card to an Oc3 port on one of the Catalyst 8510 switches. Another jumper will connect the Oc12 ports on the switches. Finally, a jumper will connect the Oc3 interface on the router to a port on one of the switches. There will be 3 free Oc3 interfaces (total) on the switches.
Standard Category-5 UTP cabling will connect the FastEthernet interfaces on the nodes to the Catalyst 2924M-XL switches. One of the switches will also connect over standard Cat-5 UTP cabling to the router. In addition, 3 Category-5 crossover UTP cables will connect the switches together, to provide the EtherChannel link at 300Mbps. There will be a total of 7 free 10/100 ports on the switches.
In addition to the connections mentioned above, initially one of the router's 10BaseT ports will be connected to NUNet via standard Cat-5 cabling. As the project to upgrade NUNet progresses, that connection will be changed to the free 100BaseT port.