High availability

This website and a couple of others are currently running on a Virtual Private Server, hosted by AWS. Unfortunately that means if it dies, all of my websites, emails and services go down. :( Because that’s not a very good way of doing things I’m creating a ‘Highly Available’ platform. AWS make this fairly easy, though it’s not without problems.

High availability means there should be duplicate failover servers, preferably loadbalanced. The problem with this is files. What happens when you modify a file on one server? There’s a couple of solutions that I looked into:

  • GlusterFS. This is a very clever thing, and can synchronise files between several servers, giving a replica on each server. The problem is that it needs a bit of setting up, and this means it doesn’t autoscale very well.
  • NFS. Linux native and pretty robust. The problem is that all files would be served from a central location. If that fails, there isn’t a local replica and failover isn’t easy.
  • Rsync. This would rely on a central node, but all of the server nodes would be able to keep a local replica. Probably the easiest option.

After some testing, rsync came out on top. It’s not without its problems though. The biggest to overcome is file uploading and editing. It’s not ideal for applications where everyone is uploading files, that bit still needs some thinking.

With things like wordpress, is it possible to redirect people to the central server for /wp-admin ? After a fair bit of reading around, it turned out that AWS make this very easy. By having the website behind CloudFront, rules can be setup so that wp-admin/ and wp-login are pointed to the central server, and other things are directed to the load balancer. It also allows static content (pictures, css files etc) to be servers from edge locations for faster downloading, though that’s something to work out for another day.

Another thing that AWS provide is databases with automatic failover to another availability zone. All I have to do is check a box, and the rest is done for me.

An interesting project

Some time ago I was working on a project to update a phone network. The phones were all Cisco 79XX range (15 7941 and 1 7970), and after some reading we found that we could create custom menus, directories and images. The directories and menus are all XMLs that it gets from a web server, so of course the logical thing to do is to make a database backend (everyone loves databases).
phone-db

This allowed us to show different menus based on the phone’s MAC. A bit of PHP later and we had a working system, with a nice front end.

The other part that we wanted to make happen was pictures. The phones allow PNGs as background graphics, but menu images were a bit more complicated. Finding documentation for Cisco can be a bit hit and miss, but we found the bits we needed. The images are 2 bit greyscale, and sent as a string of hexadecimal values. The phone also requires you to know the height and widthof the image, and every block of 4 pixels is reversed:

phone-imgIn the end the process used to make images happen was with another PHP script to do some processing:

  • Scale the image to the correct size for the phones
  • Iterate through each pixel, read the RGB value, and convert to 2 bit greyscale
  • Every 4 pixels, the order of bit pairs were reversed and put into an output string.
  • Convert to Hex, ensuring that the output of each byte was 2 characters long (otherwise 00000000 would become 0x0, and the phones don’t like it when parts are missing)

With that done we started putting static images into the menus, like llamas.

llamaIt may need some work to make it look better, but we were happy with that. For about 5 minutes. Then we decided that we could use the script to put live webcam pictures on the phones.

webcamIf you really want a network phone that can display pretty pictures, go for a more modern one that can deal with normal image files (it’s just not worth it).  This whole thing took far too long to implement, but it was a fun journey.

EMF

Recently I went to EMF a hacker/maker camp in Milton Keynes. Not content enough with all of the talks and workshops going on, me and a couple of buddies decided to make a radio station.

EMFM_sign.jpg

Unfortunately, due to licensing issues, we weren’t able to broadcast on FM. Online was good enough though, and we had streaming servers on- and off-site.

EMFM_studio.jpg

Our ‘studio’ could handle a host and 4 guests, a band, and 2 playout computers, running different softwares. The main broadcast desk was an A&H XB14, meaning that we could easily have monitors that muted when the mics are turned up.

btb_back.jpgbtb_front

 

 

 

 

 

The audio output from theXB14 went into a magic box (we haven’t come up with a good name yet). Inside the enclosure is a Raspberry Pi B with a Wolfson audio card, hooked up to a small screen and some XLR input/output. A bit of liquid soap later, and it was streaming to the onsite VPS, running icecast. An offsite VPS was running another icecast instance, relaying all of the mountpoints from the onsite server.

biz_card.jpg

I also made some friends, including Just Add Sharks, who sell laser cutters. Having a couple of working lasers just opposite the radio tent was quite interesting, and I ended up making some snazzy business cards, cut from 3mm laser ply. Everyone seems impressed so far.

emf_duck.jpgemf_iss.jpgemf_laser.jpg

Other oddities around the campsite included a motorised duck, an arrow that pointed at the International Space Station, and a guy who builds his own lasers. We even managed to burn a hole in a wetherspoons reciept I had in my pocket.

servo_arm.jpg

Oh, and I bought a robotic arm. Which I built on the way home, while waiting for a bus.

Keynestock 2014

Every year at the University of Kent there’s a small(ish) music competition called Keynestock (it happens in Keynes college). It’s a fairly interesting gig, because I like to get stuck in with everything. I put my Kent Tech hat on and organise the stage, lighting and sound. Then I put on my CSRFM hat, and help with the radio outside broadcast. After that is the KTV hat, and give them a hand with the TV broadcast. It’ a nice day, and everyone gets to try out some new stuff without being too far from home.

keynestock1

keynestock2

 

Kent Tech’s end of things went very well. For lighting we had 16 Chauvet LED battens, 4 Mac 250, 2 DTS Nick 1201, 4 molefay, 8 LED par cans and a Chamsys PC wing for control. Sound was D&B, 2 each of C4 subs and tops per side, and a B2 in the centre. 4 Max 15 monitors across the front, and a half stack of HK Actor for drum monitors. Control was from 2 SI Compact32 and a digital multicore, though not in the normal monitor/FOH configuration. One desk was at FOH, controlling monitors and FOH. The other was for the broadcast mix.

keynestock3

Student media had a slightly more interesting time of things. Our normal network had been cut off by configuration changes, and no one is ever about at 5pm on a Friday to fix it. CSRfm and KTV both rely on a network connection to stream OB outputs to base, for being sent out to the world. The fix was to install a temporary point to point wifi link using a pair of Nano Bridge M5’s, and extend the Student Media Centre network.

keynestock4

KTV probably had the most interesting time, because they wanted to have a base studio at the Student Media Centre. The OB site had 3 cameras; a jib (part of their new kit, I even played with it for a while), an onstage camera, and a fixed camera. These all fed into a BlackMagic vision mixer, which was streamed to base with a Terradek cube. The base studio had 2 cameras, vision mixer, and an over the top sound desk. The output there was streamed direct to youtube for all to watch.

keynestock5

CSR’s new toys were a pair of Raspberry Pi’s with Wolfson audio cards, used to encode and decode the audio output and send it over an IP network. OpenOB worked very well, allowing us to send very high quality audio, though we weren’t able to send a return stream (a raspberry pi doesn’t quite have a processing power).

An interesting Chrome plugin

I came across an interesting chrome plugin called Dictionary of Numbers. This compares numbers with real life values. It’ll tell you that 30 degrees Celsius is around the melting point of butter, and that 27mph is the top speed that Usain Bolt reached on his 100m world record sprint. And it inserts this information into a web page, so that you can know that 100 acres is about the size of Vatican City.
number

A cautionary tale

DNS makes the internet work. For the uninitiated, it changes human readable domain names (such as www.ciaran.com) into an address that a computer can understand (like 54.76.238.251).

A little while ago I moved this website onto a VPS. For the most part, it went fairly well. Changed the DNS entries, and waited a little while for it to go through. Once it had, I noticed that I could access “www.ciaran.com”, but “ciaran.com” always gave a 403 ‘forbidden’ error. After a couple of hours of changing apache config files with absolutely no change, I decided to ping each of them. The answer to this problem was that the DNS had been cached for “ciaran.com” at the old IP address, but “www.ciaran.com” had the new address. AAAARRRRGGGGG! After flushing the cache everything worked happily.

So the moral of the story: Always flush your DNS cache. Always.

Student Media Centre

A few weeks ago we finished converting the Kent Union ‘Attic’ into a brand new Student Media Centre. The attic was a small live music and bar area above the Venue, the campus nightclub. Several rooms were put in, including a radio studio, tv studio, an office and meeting rooms. Several hundred metres of cable was installed between the Central Technical Area (CTA) and most rooms, comprising of 10 pair, 5 pair and 1 pair audio cable, starquad, multicore coaxial and single core coaxial cables. Oh, and tonnes of Cat5.Student Media Centre

The radio studio was fitted with a Sonifex S2 split mixer, 4 Rode microphones, 2 Numark CD players, a Soundweb Blu-100 digital signal processor, and 5 small form factor computers. Why bother with a digital signal processor? An audio sensitive space above a nightclub isn’t the best of ideas, and frequencies above about 350Hz can be heard in the studio. So using a DSP unit allows us to change the EQ, gate and compression when the club is operating. Wooo! Playout is provided by a computer in CTA, with 2 remote KVM units allowing use of the screens, mice and keyboards from the studio.Radio Studio

The ‘Performance Space’ is a multipurpose space, with a 32/16 audio multicore, and two 6 way coaxial multicores, all connecting back to the Gallery. In the Gallery there’s a Behringer X32 audio desk, a Black Magic vision mixer, and several computers. Audio from the X32 is fed into the vision mixer via AES, and the analogue outputs are sent to the two monitor speakers mounted on the wall. A third speaker is sent the monitor feed, so that using PFL doesn’t scare the producer, and the main output is always audible. Video is sent via coax to a dedicated streaming computer in CTA with a BlackMagic Decklink card, which sucks in video and sends it out to the internet (usually youtube).TV Gallery32/16 Multicore

 

 

 

 

CTA has a mix of equipment, with a UPS, 3 computers, 5 Xnodes, 3 Cisco switches, and a Wharton master clock. The computers serve as audio playout for the radio studio, video streaming, and an Mac server. The switches provide network and PoE for all the computers, CCTV cameras and VoIP phones located about the building. Lastly, the master clock receives an MSF radio time signal from the outside world. It then outputs a code for the 3 clocks in the building, and acts as an NTP server for everything to synchronise to.

Central Technical Area

CTA Krone Frames

 

 

 

 

 

 

 

Joining 3 media groups that were previously separate provided some interesting challenges. A new domain was created for Active Directory, and all previous users migrated. Storage capacity was also increased to deal with video content that had previously been dealt with by the Union. The phone system was also overhauled, with 3 new phones added, and the introduction of 3 digit extensions (instead of 2). The phones were also programmed to display images, and a system was put in place to manage phone directories and webcam images.Phones

For more pictures and details, go to www.facebook.com/StudentMediaCentre

MagicQ on a netbook?

Netbooks aren’t really the best type of computer to run lighting software. My Samsung N150 isn’t very powerful by today’s standards, with a 1.6GHz Atom processor, 1GB RAM and a 10.1″ screen (That’s a very small screen).

The problem is that Windows is quite heavy, and most lighting softwares can cripple the netbook. As well as that, the taskbar and window decorations take up a huge percentage of screen space, enough to make many programs unusable. So, what do? Chamsys MagicQ can be run on Linux, Windows and Mac, and is pretty light weight (compared to the likes of Avolites and MA), so this seemed the best route to go down.

Stability was top of the list for requirements, so I decided to make a separate installation of Debian Wheezy (Stable), and keep it well away from my dev instance. I also wanted a lightweight, minimal window manager to run it in. RatPoison seemed to fit the bill, a tiling window manager with no window decoration or task/menu bars.

After getting a version of MagicQ on the netbook (I used elinks, a console web browser), there were a couple of libraries to install. Finally, I set up X11 to start automatically, and get logged in. The ~/.xinitrc file also had to be edited a bit:

#~/.xinitrc

#!/bin/sh

# Set the keyboard to GB
setxkbmap gb

# Disable screensaver and disable screen blanking
xset s off
xset s noblank

# Start windowmanager
ratpoison &

# Launch MagicQ
cd ~/magicq
./magicq

#Shutdown when finished
sudo /sbin/shutdown now -h -t 0

So The comments explain most of it. Because each item is only executed after the previous one is finished (unless followed by an ampersand & ), the shutdown command is only executed after MagicQ is closed or killed. This means that the netbook is basically shutdown by the MagicQ software. It does need a little bit of tweaking of the sudoers file, to allow ‘shutdown’ to be run without asking for a password.

username ALL = (root) NOPASSWD: /sbin/shutdown

Remember to use visudo instead of manually editing the /etc/sudoers file.

So far, this setup hasn’t ever crashed, frozen or died on me, even when doing pixel mapping!