Raspberry pi cluster what is it. Parallel computing on raspberry pi. The concept of a cluster and cluster computing
One popular use of Raspberry Pi computers is building clusters. Raspberry Pies are small and inexpensive so it"s easier to use them to build a cluster than it would be with PCs. A cluster of Raspberry Pies would have to be quite large to compete with a single PC; you"d probably need about 20 Pies to produce a cluster with as much computing power as a PC. Although a Pi cluster may not be that powerful, it's a great opportunity to learn about distributed computing.
There are several different types of distributed computer which can be used for different purposes. There are super computers that are used for solving mathematical problems like modeling weather conditions or simulating chemical reactions. These systems often use the Message Passing Interface (MPI). A team at the University of Southampton built a 64 node MPI based super computer . This system is used for teaching students about supercomputing.
Another technology that"s often used in distributed computing is Hadoop, which distributes data across many nodes. Hadoop is often used for processing large datasets and data mining. An engineer at Nvidia built a small Hadoop cluster using Raspberry Pies . He uses his cluster to experiment and test ideas before deploying them on more powerful systems.
Using a Raspberry Pi cluster as a web server
Clusters can be used as web servers. Many web sites get too much traffic to run on a single server, so several servers have to be used. Requests from web browsers are received by a node called a load balancer, which forwards requests to worker servers. The load balancer then forwards responses from servers back to the clients.
This site is now hosted on a Raspberry Pi cluster. The worker nodes are standard web servers that contain identical content. I just installed Apache on them and copied my site to each node.
I use an extra Raspberry Pi to host a development copy of this site, and to control the cluster. This Pi is connected to my local network via wifi, so I can access the development copy of my site from my laptop.
The extra Pi also has an ethernet connection to the Pi cluster. When I want to update my site, I can transfer changes from the development site to the live site on the cluster. Site updates are put into .tar.gz files which the worker nodes automatically download from the development site. Once downloaded, updates are then unpacked into the local file system.
Configuring the Raspberry Pi servers
All of the Pies in this system are headless. I can log into the Pi with the development site using the Remote Desktop Protocol, and from that Pi I can log into the worker Pies using SSH.
All the Pies in the cluster use a static IP address. In a larger cluster it would probably be better to set up a DHCP server on the load balancer. The IP addresses used in the cluster are on the 192.168.1.xxx subnet.
For each worker Pi, I set up a 4GB SD card using the latest version of Raspbian. In raspi-config I set the following options:
- expand fs
- set the hostname
- set the password
- set memory split to 16MB for the GPU
- overclock the CPU to 800MHz
- enable ssh
On each card I installed Apache and some libraries required by my CMS, libxml2 and python-libxml2. I used this command to enable mod rewrite, which is also required by my CMS:
$ sudo a2enmod rewrite
Finally, I copied some scripts onto each SD card which allow each Pi to synchronize its contents with the development Pi. In a larger cluster it would be worth creating an SD card image with all of these modifications made in advance.
Building a load balancer
The load balancer must have two network interfaces, one to receive requests from a router, and another network interface to forward requests to the server cluster. The nodes in the cluster are a on a different subnet than the rest of the network, so the IP address of the load balancer"s second interface must be on the same subnet as the rest of the cluster. The load balancer"s first interface has IP address 192.168.0.3 while the second interface"s IP address is 192.168.1.1. All the Pies in the cluster have IP addresses on the 192.168.1.xxx subnet.
I built my load balancer using an old PC with 512MB of RAM and a 2.7GHz x86 CPU. I added a second PCI ethernet card and installed Lubuntu, a lightweight version of Ubuntu. I was going to install Ubuntu, but this PC is pretty old, so Lubuntu is probably a better choice. I used a PC becasue I wasn't sure if a single Pi would be powerful enough to act as a load balancer, and a Pi only has one ethernet connection. I want both of my load balancer's network connections to be ethernet for improved performance and stability.
Note that IP forwarding is not enabled. The load balancer isn't a router, it should only forward HTTP requests and not every IP packet that it receives.
Setting up the load balancer software
There are many different software implementations of load balancing. I used Apache"s load balancer module because it"s easy to set up. First I made sure my PC's OS was up to date:
sudo apt-get update
sudo apt-get upgrade
Then I installed Apache:
sudo apt-get install apache2
These Apache modules need to be enabled:
sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod proxy_balancer
The next step is to edit /etc/apache2/sites-available/default in order to configure the load balancer. The proxy module is needed for HTTP forwarding, but it's best not to allow your server to behave as a proxy. Spammers and hackers often use other people's proxy servers to hide their IP address, so it's important to disable this feature by adding this line:
ProxyRequests off
Although proxy requests are disabled, the proxy module is still enabled and acts as a reverse proxy. Next, define the cluster and its members by adding this code:
Balancer manager interface
The balancer module has a web interface that makes it possible to monitor the status of the back end servers, and configure their settings. You can enable the web interface by adding this code to /etc/apache2/sites-available/default:
It"s also necessary to instruct Apache to handle requests to the /balancer-manager page locally instead of forwarding these requests to a worker server. All other requests are forwarded to the cluster defined above.
ProxyPass /balancer-manager ! ProxyPass / balancer://rpicluster/
Once these changes have been saved, Apache should be restarted with this command:
$ sudo /etc/init.d/apache2 restart
when I open a browser and go to http://192.168.0.3 I see the front page of my web site. If I go to http://192.168.0.3/balancer-manager, I see this page in the image on the right.
The last step in getting the cluster online is adjusting the port forwarding settings in my router. I just needed to set up a rule for forwarding HTTP packets to http://192.168.0.3.
Here"s the complete /etc/apache2/sites-available/default for the load balancer:
Used by enthusiasts for a variety of purposes. For example, enthusiast David Guill decided to use it to build a cluster - a group of computers connected to each other and representing, from the user's point of view, a single hardware resource. It was decided to name the project 40-Node Raspi Cluster. It is worth noting that David needed a cluster in order to gain experience in programming on distributed systems, so the Raspberry Pi cluster will replace, for the duration of the training, a real super-computer.
In a simple way, it would be possible to assemble a cluster from a Raspberry Pi, using a rack or an inexpensive cabinet instead of a case (as in the case with), but since David is fond of modding, he decided to make a stylish cluster, as close as possible in appearance and convenience to serial solutions. And, I must say, David succeeded, because his project is much more thought out than many serial cases. By the way, the case of the 40-Node Raspi Cluster project is made of acrylic panels, cut to size with a laser and glued by hand.
The main distinguishing features of the 40-Node Raspi Cluster project are: a cool appearance, a fairly compact size (like a large tower), easy access to all components and the ability to replace them without having to disassemble the case, screwless fastening of parts of the case and many components, as well as order in wires (and there are quite a few of them in this project). This project includes 40 compact Raspberry Pi computers (40 Broadcom BCM2835 cores with a frequency of 700 MHz, 20 GB of distributed RAM), two 24-port switches, one ATX form factor power supply, five 1 TB hard drives (with expandable up to 12 pieces), 440 GB of flash memory, as well as a router with wireless connectivity.
The Raspberry Pi computers in this project are grouped by four on custom acrylic mounts, of which there are ten in this project. Thanks to this mount (as in blade servers), convenient access and easy replacement of compact computers is provided. Each blade with Raspberry Pi has its own compact DC-DC converter, powered by a common ATX power supply. The cluster is cooled by four 140 mm fans behind which filters are installed.
An abundance of LEDs adds additional “modding” to the 40-Node Raspi Cluster project - there are more than three hundred of them in the project (on mini-computers, switches, routers and fans), while during operation they will blink in accordance with the load. The dimensions of this project are 25 x 39 x 55 cm, and the approximate cost of construction is $ 3,000.
With the external and internal appearance, as well as with the features of the 40-Node Raspi Cluster project, you can see the attached photos and videos. If you are interested in this project, then get acquainted with it in more detail, and David described the construction of this monster in great detail, you can visit the corresponding note on his personal website.
In today’s guest post, Bruce Tulloch, CEO and Managing Director of BitScope Designs , discusses the uses of cluster computing with the Raspberry Pi, and the recent pilot of the Los Alamos National Laboratory 3000-Pi cluster built with the BitScope Blade .
High-performance computing and Raspberry Pi are not normally uttered in the same breath, but Los Alamos National Laboratory is building a Raspberry Pi cluster with 3000 cores as a pilot before scaling up to 40,000 cores or more next year.
That's amazing, but why?
Traditional Raspberry Pi clusters
Like most people, we love a good cluster! People have been building them with Raspberry Pi since the beginning, because it's inexpensive, educational, and fun . They’ve been built with the original Pi, Pi 2, Pi 3, and even the Pi Zero, but none of these clusters have proven to be particularly practical.
That's not stopped them being useful though! I saw quite a few Raspberry Pi clusters at the conference last week.
One tiny one that caught my eye was from the people at openio.io , who used a small Raspberry Pi Zero W cluster to demonstrate their scalable software-defined object storage platform , which on big machines is used to manage petabytes of data, but which is so lightweight that it runs just fine on this:
There was another appealing example at the ARM booth, where the Berkeley Labs’ singularity container platform was demonstrated running very effectively on a small cluster built with Raspberry Pi 3s.
My show favorite was from the Edinburgh Parallel Computing Center (EPCC): Nick Brown used a cluster of Pi 3s to explain supercomputers to kids with an engaging interactive application. The idea was that visitors to the stand design an aircraft wing, simulate it across the cluster, and work out whether an aircraft that uses the new wing could fly from Edinburgh to New York on a full tank of fuel. Mine made it, fortunately!
Next generation Raspberry Pi clusters
We’ve been building small-scale industrial-strength Raspberry Pi clusters for a while now with BitScope Blade .
When Los Alamos National Laboratory approached us via HPC provider SICORP with a request to build a cluster comprising many thousands of nodes, we considered all the options very carefully. It needed to be dense, reliable, low-power, and easy to configure and to build. It did not need to “do science”, but it did need to work in almost every other way as a full-scale HPC cluster would.
I'll read and respond to your thoughts in the comments below this post too.
Editor's note:
Here is a photo of Bruce wearing a jetpack. Cool, right?!
16 comments
What’s wrong with using the switch and ethernet fabric as the “bus” on the existing hardware?
That's a good idea. I think the time is right.
The EPCC Raspberry Pi Cluster is called Wee Archie () and it (like the Los Alamos one we built) is a “model”, albeit for a somewhat different purpose. In their case it’s representative of Archer (http://www.archer.ac.uk/) a world-class supercomputer located and run in the UK the National Supercomputing Service. Nick Brown (https://www.epcc.ed.ac.uk/about/staff/dr-nick-brown) is the guy behind the demo I saw at SC17. Drop him a line!
One of the pending issues with exascale computing is that it is inefficient to checkpoint a computation running on so many cores across so many boxes. At the same time, the probability that all nodes function faultlessly for the duration of the computation decreases exponentially as more nodes are added.
Effectively utilizing distributed memory parallel systems has been compared to herding chickens. When contemplating flocks so large that it takes megawatts to feed them, it may be better to practice by herding cockroaches. This isn't about performance tuning Fortran codes, but how to manage hardware faults in a massively distributed parallel computation. As mentioned in the press release, we don't even know how to boot an exascale machine: By the time the last node boots, multiple other nodes have already crashed. In my opinion modeling these exascale difficulties with a massive cluster of Raspberry Pi computers is feasible. For example, dumping 1GB of RAM over the Pi's 100Mbit networking is a similar data to bandwidth ratio as dumping 1TB of RAM over a 100Gbit interconnect.
Spot on Eric. The issue is one of scale, booting, running the machines, getting the data in and out and check-pointing to avoid losing massive amounts of computational work.
Some interesting things I learned from this project…
One normally thinks of error rates of the order of 10^-18 as being pretty good, but at this scale one can run into them within the duration of a single shot on a big machine. At exascale this will be worse. The word the HPC community uses for this is “resilience”; the machines need to be able to do the science in a reliable and verifiable way despite these “real world” problems intervening in the operation of the underlying cluster.
They do a lot of “synchronous science” at massive scale so the need for check-points is unavoidable and Los Alamos is located at quite a high altitude (about 7,300 feet) so the machines are subject to a higher levels of cosmic radiation. This means they encounter higher rates of “analog errors” which can cause computation errors and random node crashes.
All these sorts of problems can be modelled, tested and understood using the Raspberry Pi Cluster at much lower cost and lower power than on big machines. Having root access to a 40,000 core cluster for extended periods of time is like a dream come true for the guys who's job is to solve these problems.
Not LFS but not Raspbian either (except for initial testing). They will eventually published more to explain what they’re doing but suffice to say it’s a very lean software stack which is aimed to make it easy for them to simulate the operation of big clusters on this “little” one.
When running a cluster of 750 nodes (as Los Alamos are doing), managing and updating images on all 750 SD card is, well, a nightmare.
If your cluster is smaller it may not be a problem (indeed we often do just this for small Blade Racks of 20 or 40 nodes).
However, the other issue is robustness.
SD cards tend to wear out (how fast depends on how you use them). PXE (net) boot nodes do not wear out. Local storage may also not be necessary (if you use an NFS or NBD served file system via the LAN) but the bandwidth of accessing the (remote) storage may be a problem (if they all the nodes jump on the LAN at once depending on your network fabric and/or NFS/NBD server bandwidth).
The other option are USB sticks (plugged into the USB ports of the Raspberry Pi). They are (usually) faster and (can be) more reliable than SD cards and you can boot from them too!
All that said, there is no problem with SD cards used within their limitations in Raspberry Pi Clusters.
You guys need to start making a computer “erector set”. As a kid, I received an erector set when I was 7-8 years old for Christmas. I made everything one could think of. I later became a Mechanical Engineer. I designed parts for GE Gas Turbines, and when you switch on your lights, I have a direct connection to powering RPis all over the world.
You have most of the fundamental parts right now. You need a bus, something like the CM DDR3 bus. If the RPi 3B or whenever the RPi 4 comes out, had an adapter or pinouts that connected to that bus, Clustering would be easy. I could envision four quad processor CMs, a graphics processor/Bitcoin miner on a CM, a CM with SSD, etc. A computer erector set…
Is there a short video presentation available that discusses the Los Alamos Pi cluster, how it was constructed, what it will be used for and why this solution was chosen over others?
Also, given the interest in OctoPi and other Pi clusters, could there be a section devoted to parallel processing in the Raspberry Pi Forum?
Is the airwing demo available?
I'm glad I left their high performance computing department now. This is madness. The Fortran code bad so prevalent at the labs is not going to run the same on the ARM architecture when the super computers the code is to run on will be used on Intel architecture machines. This project is going to give the interns a playing field to learn what they should have learned in college.
I make 120 Raspberry Pi clusters for 3D scanning. Use pure UDP multicasting to control them all using a single network packet transmission. Works really well :-)
That's very similar to what we want with a new grass roots project. But instead of a cluster physical near located, We are thinking of a ‘collective’ (kind of Borg, but then nice…), for doing Three.js GPU 3D rendering. I’ve got a prototype running on If you Bing or Google on sustasphere, you will find the corresponding GitHub (not completely up to date however). The current prototype renders (obviously) in your browser. With the collective, your browser-calls will be routed to (hopefully) thousands of Raspberry’s; each crunching real-time a part of the 3D rendering. In ‘my head’, I’m thinking about Open Suze stacked with Express.js.
For the energy-supply of each node, we thank the wind and an archemedian screw, with an hydraulic head, with a simple bicycle dynamo…
Nice, but why? We would like to honor an echo from the past (The Port Huron Statement); introduce a virtual sphere of dignity. Giving people the opportunity to express their emotions; again defining the meaning of dignity. Imagine Mozart performing his Nozze di Figaro (for me a perfect example of bringing art to the people and sharing thoughts about morality); and being able to actually be there, move around, ‘count the nostrils’ and maybe even ‘get physical’.
Yep, you'll need some GPU-collective for that.
Based on your experience, could you advise us on our road ahead? Help use make sound decisions?
> recent pilot of the Los Alamos National Laboratory 3000-Pi cluster
It should read 750-Pi cluster, 5 blades of 150 Pis each, with 3000 cores total (4 cores each per CPU)
Ok, I'm a nuby on raspberry pi 3's. But I was wondering if they used LFS with the bitscope blade cluster? …and if so, how did it perform?
Why is it “important to avoid using the Micro SD cards” ?
I have an application in mind for a pi cluster, for which I will need local storage. If I can't use the MicroSD card, then what?
Introduction
The fundamental costs in the field of computing are the power of computers and their energy consumption. Modern supercomputers take up huge space and consume hundreds of thousands of watts.
A big problem in this case is the process of teaching parallel programming and performing calculations on supercomputers, and even more so, managing this type of computers, since, as a rule, students do not have direct access to them.
In this paper, it is proposed to solve this problem using single-board microcomputers that have recently appeared on the computer equipment market (Paspberry Pi and analogues). Based on them, you can assemble an inexpensive computing cluster and teach students the basics of parallel programming. Thus, the purpose of this work is to create an inexpensive educational cluster of microcomputers for the development and implementation of parallel programming algorithms in the educational process. An example of parallel computing in the developed training cluster is demonstrated.
The Raspberry Pi is a microcomputer developed by the Raspberry Pi Foundation. Small, the size of a bank card, it is a full-fledged single-board computer (System- on- a- chip). Processor (in PI 3 model): 4 cores ARM Cortex-A53x64. The default operating system is Raspberian (based on the Linux kernel). At a price of only $35, the board has all the necessary interfaces ( Wi- fi, Bluetooth, Usb, ethernet), as well as a large set of ready-made programs for any kind of activity. That is why these microcomputers were chosen for a small educational computing cluster.
The concept of a cluster and cluster computing
It is well known that a cluster is a group of computers connected by high-speed communication channels, representing a single hardware resource from the user's point of view. On the other hand, a cluster is a loosely coupled collection of multiple computing systems working together to execute common software applications. In order to link several raspberry PIs into a cluster, a typical cluster computing system (router, Ethernet cables, USB, etc.) was assembled based on the PI 3 processor (Fig. 1).
Figure 1. Processor-based cluster computing system of two PI 3
Demonstration of parallel computing
For a visual demonstration of the capabilities of a cluster of two PI 3, the Python 2 programming environment and the implementation of the array sorting algorithm by the merge method were chosen. The computers were connected by a local network. To simplify clustering of multiple R PIs, there are many ready-made programs, one of which is called “mpi4py” .
The merge sort array code in Python looks like this:
def merge(left,right): #merges 2 sorted lists together
#Goes through both lists
while i< len(left)and j < len(right):
#Adds smaller element of the lists to the final list
if left[i]<= right[j]:
result.append(left[i])
result.append(right[j])
result += left
result += right
mergesort(lst):
#if there"s only 1 element, no need to sort
if len(lst)< 2:
#breaks down list into 2 halves
middle = len(lst)/ 2
#recursively splits and sorts each half
left = mergesort(lst[:middle])
right = mergesort(lst)
#merges both sorted lists together
return merge(left, right)
The algorithm of the program consists of the following sequence of actions:
1. On PI 3 (server) a random array of numbers is generated.
2. This array is divided into n parts, according to the number of processors in the local network.
3. Using the socket module and the local network, the Pi3 (server) transmits part of the Pi3 array (client).
4. Pi3 (server) sorts its part of the array and waits for Pi3 (client) to respond.
5. Pi3 (client) sorts its part of the array and sends it to Pi3 (server).
6. Pi3 (server) receives the sorted part of the array and performs the final sort.
Calculations showed that it took one Pi3 about 23 seconds to sort an array of 500 thousand elements. After adding a second Pi3, this time was reduced to 16 seconds. The increase in speed is non-linear, but the more computers there are in the cluster, the less time will be spent.
Conclusion
Single-board computers have only recently moved beyond the factory automation segment and started to conquer the mass market. Their small size, low power consumption and sufficiently high computing capabilities can make them the basis for the implementation of various projects, for example, teaching parallel programming. A feature of the presented cluster computing system based on raspberry PI 3 is good scalability, determined by the capabilities of switching equipment, low cost, the ability to use free software, which is important when introducing it into the educational process. The demo work carried out shows that a cluster of even two PI 3 is able to speed up the calculation of a simple, but at the same time voluminous task, such as sorting a large array of data.
In the future, it is planned to increase the number of microcomputers in the computing system and compare the performance of cryptographic algorithms, in particular, those planned to be used to encrypt/decrypt images (photo, aerial, space images) of large volume and transfer them over the Internet.
List literature:
- Robert Mullins/ Distributed computed //University Cambridge. – 2012. – http://www.cl.cam.ac.uk/projects/raspberrypi/tutorials/distributed-computing / .
- Cluster. - Access mode. – URL: https://ru.wikipedia.org/wiki/Cluster (accessed 02/25/2017).
- Lukin V.V., Marchevsky I.K. Educational and experimental computing cluster. Part 1. Tools and capabilities. - Access mode. – URL: https://elibrary.ru/download/elibrary_17091004_33209664.pdf (accessed 02.25.2017).
A 4-node Raspberry Pi Cluster. The top board is an original Model B, while the three below are brand new Raspberry Pi 2 boards.
updated: This project was originally published on 26th Aug 2015 and was then updated on the 5th Sept 2015 with additional instructions on how to add a second Ethernet adapter to the head node, and have it serve as a DHCP server for the other nodes in the cluster.
Over the weekend I sat down and built a small Raspberry Pi cluster consisting of 4 nodes. I used three boards for compute nodes and an for the head node. I wanted the cluster - more commonly known as a 'bramble' - to be as compact as possible, with just two cables coming out, one for power and the other for network. I made use of a USB hub to power the boards, and a small Ethernet switch which I could hack to be also powered from the USB hub rather from a separate wall wart.
It's hardly the biggest cluster built from Raspberry Pi boards, as far as I know the 120 Pi cluster built by the folks at Resin.io is still the biggest built, although since it's actually 5 independent 24-node clusters, possibly the still takes the title.
However, even with just 4 nodes my pocket cluster is big enough for what I want, which is as a testbed for some distributed computing work I’m doing. The small cluster sitting on my desk lets me test code out before deploying jobs to the much more extensive, and expensive, cluster I’m using for grunt work on the project.
The 4 board ‘dogbone’ enclosure
The enclosure I finally settled on was a four board stackable ‘dog bone’ case that I picked up on Amazon , although if you’re willing to wait a little bit there are plenty of similar cases on AliExpress that can be had for much less. It shipped overnight and I had it the next day; it was the only thing I bought to build the cluster as I had everything else on the shelf.
The 5-port USB Hub
The USB hub I used was the thing that actually inspired me to do the build in the first place: It’s a 5-port hub from Anker and has coincidentally about the same footprint as the Raspberry Pi itself. With five ports, there's one port for each of my four Raspberry Pi boards, and a final port left over to power an Ethernet switch for the cluster.
The 5V power supply and a spare USB cable
The first step is to carefully snip off the end of the 5V supply cable, making sure to label which of the two wires corresponded to the two wires left attached to the power brick. Stripping off the ends of the wires you can plug the brick into the wall and use a volt meter to measure which of the two wires is +5V and which is GND.
The 5V supply cable (top) and the USB cable end (bottom)
Then snip off the end of the USB cable and carefully, as the wires inside the cable are small and delicate, strip back the cover to reveal the wires. You're looking for the red and black wires, the others carry data. You can just cut them off, you won't need them.
The internal wiring of a USB cable
Soldering the two end of the cables together - joining the +5V to +5V, and the GND to GND - and then covering each individual wire, as well as the join itself, with some shrink wrap gives me the Frankenstein cable I need to power the Ethernet switch from the last available port of my USB hub.
The Frankenstein cable
After searching through my stack of spare cables to find the shortest USB and Ethernet cables possible, sticking the cluster together at this point came down to cable ties and velcro.
The finished Raspberry Pi cluster
% sudo apt-get install autofs
and then edit the /etc/auto.master file adding
/mnt/nfs /etc/auto.nfs
at the end. Then create the /etc/auto.nfs file, adding,
rpi0 rpi0:/mnt/usb
and restart the autofs service,
% sudo /etc/init.d/autofs restart.
if all goes well at this point if you change to the /mnt/nfs/rpi0/ directory and the disk attached to the head node should automatically mount itself. You can check
%df -h Filesystem 1K-blocksUsed Available Use% Mounted onrootfs14984668 25132281181235618% //dev/root 14984668 25132281181235618% /devtmpfs470416 0470416 0% /dev tmpfs94944 232 94712 1% /run tmpfs 5120 05120 0% /run/lock tmpfs 189880 0189880 0% /run/shm/dev/mmcblk0p1 57288 19448 3784034% /bootrpi0:/mnt/usb 604670086460466944 1% /mnt/nfs/rpi0
to see whether it has been automatically mounted.
Blinking Lights
Alongside the USB flash drive (since I had one lying around) I installed a Blinkstick. A single software-controllable RGB LED, the stick actually comes in rather handy for server status light. It's hard to ignore a blinking light. After slotting the stick into the head node’s last remaining USB port, you can set up the software by,
% sudo apt-get install -y python-pip python2.7-dev % sudo pip install blinkstick % sudo blinkstick --add-udev-rule
from there it's actually pretty easy to manipulate the RGB LED from the command line.
Or when you want to use the Blinkstick programmatically to indicate status you can use the API, and your programming language of choice .
Next steps
I travel a lot. That means I spend a lot of time away from my home office. While I can leave the cluster up and running and just ssh into it while I’m away, I’d actually sort of like to be able to take it on the road with me to shows. So, going forward, I'd really like just to be able to pick the cluster up and dump it down on any network.
That means I'm going to have to reconfigure the networking just a little bit.
Instead of directly connecting the Ethernet switch to the external network, and having my home router allocate IP addresses for each of the nodes, as a next step I’m going to add a USB Ethernet adapter to the head node. This will give the head node two Ethernet connections.
The first will connect to the external network, giving the head node - and hence the cluster - an ‘external’ IP address. The second will connect to the cluster's Ethernet switch. We can then configure the head node as a DHCP server for other three ‘internal’ nodes attached to the switch, creating a second network visible only to the cluster.
In this configuration I’ll still be able to ssh into the head node, but I’ll only be able to reach the three compute nodes from the head node. There is a problem however: How will I know the external IP address of the head node?
Adding an LCD
The Blinkstick is good for simple messaging, you can actually do a lot with an RGB LED to let yourself know what’s going odd. But it's actually pretty easy to add a simple LCD display to the head node.
After connecting the panel you'll need to install the I2C tools and associated Python libraries
% sudo apt-get install python-smbus % sudo apt-get install i2c-tools
and to enable I2C on the head node by adding the following at the bottom of the /boot/config file,
device_tree=dtparam=spi=on dtparam=i2c1=on dtoverlay=w1-gpio-pullup,gpiopin=3,pullup=3dtoverlay=w1-gpio-pullup,gpiopin=5,pullup=5and adding the following modules to the /etc/modules file,
I2c_dev i2c_bcm2708
After rebooting the head node you should be able to see the panel with an I2C ID of 27,
% sudo i2cdetect -y 1 0123456789abcdef 00:-- -- -- -- -- -- -- -- -- -- -- -- -- 10: -- -- -- -- -- -- -- -- -- -- -- UU -- -- -- --20: -- -- -- -- -- -- -- 27 -- -- -- -- -- -- -- -- 30: -- -- -- -- -- -- -- -- -- -- -- UU -- -- -- --40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 70: -- -- -- -- -- -- -- --You can see that eth0 has the static internal IP address we allocated to it, while eth1 has a new IP address allocated by our home router. If all goes to plan you should be able to ssh into the head node using its new external IP address, and see something like this,
%ifconfig eth0Link encap:EthernetHWaddr b8:27:eb:22:60:fbinet addr:192.168.50.1Bcast:192.168.50.255Mask:255.255.255.0RX packets:2470 errors:0 dropped:0 overruns:0 frame:0TX packets:2267 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:215730 (210.6 KiB)TX bytes:237032 (231.4 KiB)eth1Link encap:EthernetHWaddr ac:29:3a:da:74:37inet addr:192.168.1.194Bcast:192.168.1.255Mask:255.255.255.0UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1RX packets:15245 errors:0 dropped:1 overruns:0 frame:0TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1787746 (1.7 MiB)TX bytes:283761 (277.1 KiB) loLink encap:Local Loopback inet addr:127.0.0.1Mask:255.0.0.0UP LOOPBACK RUNNINGMTU:65536Metric:1RX packets:4 errors:0 dropped:0 overruns:0 frame:0TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:260 (260.0B)TX bytes:260 (260.0B)% route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface0.0.0.0 192.168.1.254 0.0.0.0 UG000 eth1192.168.1.0 0.0.0.0 255.255.255.0 U 000 eth1192.168.50.00.0.0.0 255.255.255.0 U 000 eth0
If not everything goes to plan and you're stuck unable to reach the head node over the network, it's possible that you might have to dig out a HDMI monitor and a USB keyboard and connect them directly to the head node - you can temporarily yank the USB disk to give yourself and free USB port for the keyboard - so you can diagnose and fix any networking issues.
Hopefully, however, you can reach the head node from the external network. You should be able to ping both external hosts on the 192.168.1.* network, and internal hosts on the 192.168.50.* network.
However, at least right now, if we ssh into one of the compute nodes, while they can see the head node - and each other - they can’t yet see the outside world. We're going to have to forward packets from the internal to the external networks before that's possible.
On the head node go ahead and,
% sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"and then edit the /etc/sysctl.conf file uncommenting the line saying,
Net.ipv4.ip_forward=1
After activating forwarding we'll need to configure iptables ,
% sudo iptables - t nat - A POSTROUTING - o eth1 - j MASQUERADE eth1-j ACCEPT % sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"and then add at the bottom of the /etc/network/interfaces file a line to load the tables on boot,
Up iptables-restore< /etc/iptables.ipv4.nat
Rebooting the head node at this point, you should now be able to ssh into any of the compute nodes from the head node and be able to ping the outside world,
%ssh rpi1 Linux rpi2 3.18.11-v7+ #781 SMP PREEMPT Tue Apr 21 18:07:59 BST 2015 armv7lThe programs included with the Debian GNU/Linux system are free software;the exact distribution terms for each program are described in theindividual files in /usr/share/doc/*/copyright.Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Sat Sep5 20:49:07 2015 from rpi0% ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.64 bytes from 8.8.8.8: icmp_req=1 ttl=54 time=23.8 ms64 bytes from 8.8.8.8: icmp_req=2 ttl=54 time=21.4 ms64 bytes from 8.8.8.8: icmp_req=3 ttl=54 time=23.2 ms^C --- 8.8.8.8 ping statistics ---3 packets transmitted, 3 received, 0% packet loss, time 2003msrtt min/avg/max/mdev = 21.470/22.838/23.829/1.014 ms%That's it. We have a working cluster.
In Closing
At this point we have a cluster with two cables going into it, one for power and the other for network. You can plug into any network and the head node will report its external IP address on an LCD panel, allowing you to ssh into it, and from there you can ssh into - and between - any of the nodes in the cluster without needing a password . All the nodes also share a disk.
In other words, it's all pretty much all working at this point. In fact, I'm currently using it as a desktop Hadoop cluster .
From here there are a couple of things we could do, the most obvious next step would be to add some SNMP monitoring, and an external facing ‘status’ dashboard on the head node to monitor the cluster health. However in the longer term, the free Ethernet port on the switch means that we can expand the cluster fairly easily by adding another rack of four compute nodes without too much extra effort.